2024-09-16T02:16:32.821 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T02:16:32.837 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T02:16:32.908 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763 branch: main description: rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905763' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) mon_bind_msgr2: false sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPJblbmhzcwjMyiLmZgPA0g84vuWylaB01uL+2pFBi9nEd68oVqGMf85pcUCipYyfnUxn5gsxLZDthiokvRr0Tg= tasks: - install: null - ceph: conf: osd: osd min pg log entries: 5 osd_fast_fail_on_connection_refused: false log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - osd_recovery.test_incomplete_pgs: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T02:16:32.908 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-16T02:16:32.909 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-16T02:16:32.909 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T02:16:32.911 INFO:teuthology.task.internal:Checking packages... 2024-09-16T02:16:32.932 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-16T02:16:32.933 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T02:16:32.933 INFO:teuthology.packaging:ref: None 2024-09-16T02:16:32.933 INFO:teuthology.packaging:tag: None 2024-09-16T02:16:32.933 INFO:teuthology.packaging:branch: main 2024-09-16T02:16:32.933 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:16:32.933 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-09-16T02:16:33.180 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4937-g74984932-1jammy 2024-09-16T02:16:33.182 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T02:16:33.188 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T02:16:33.188 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T02:16:33.194 INFO:teuthology.task.internal:Saving configuration 2024-09-16T02:16:33.207 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T02:16:33.214 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T02:16:33.235 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 02:11:03.211090', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE+w/umyK6/W63ryB9TloQ1Pbs2c/YaYfJPBSXlXJWHN'} 2024-09-16T02:16:33.236 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T02:16:33.242 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-16T02:16:33.242 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T02:16:33.273 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1852d96e60>, signals=[15]) 2024-09-16T02:16:33.273 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T02:16:33.279 INFO:teuthology.task.internal:Opening connections... 2024-09-16T02:16:33.279 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:16:33.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:16:33.357 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T02:16:33.364 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-09-16T02:16:33.473 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-09-16T02:16:33.474 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="22.04" 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=jammy 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T02:16:33.520 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T02:16:33.521 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=jammy 2024-09-16T02:16:33.521 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-09-16T02:16:33.536 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T02:16:33.544 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T02:16:33.549 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T02:16:33.549 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-09-16T02:16:33.567 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T02:16:33.573 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T02:16:33.573 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-09-16T02:16:33.629 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T02:16:33.656 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T02:16:33.671 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T02:16:33.671 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T02:16:33.671 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T02:16:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:16:33.676 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-16T02:16:33.723 INFO:teuthology.orchestra.run.smithi137.stdout:5.15.0-56-generic 2024-09-16T02:16:33.724 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.15.0-56-generic 2024-09-16T02:16:33.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2024-09-16T02:16:33.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-09-16T02:16:33.980 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T02:16:33.999 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:16:34.005 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T02:16:34.081 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T02:16:34.234 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T02:16:34.306 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T02:16:34.334 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T02:16:34.338 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T02:16:34.341 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T02:16:34.341 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T02:16:34.341 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T02:16:34.361 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T02:16:34.363 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T02:16:34.364 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T02:16:34.369 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T02:16:34.373 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T02:16:34.382 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T02:16:34.383 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T02:16:34.383 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T02:16:34.383 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T02:16:34.412 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T02:16:34.412 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T02:16:34.416 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T02:16:34.418 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T02:16:34.419 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T02:16:34.419 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T02:16:34.439 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T02:16:34.442 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T02:16:34.442 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T02:16:34.450 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T02:16:34.457 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T02:16:34.459 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T02:16:34.459 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T02:16:34.460 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T02:16:34.475 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T02:16:34.476 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T02:16:34.476 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T02:16:34.476 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T02:16:34.477 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T02:16:34.487 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T02:16:34.487 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T02:16:34.487 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T02:16:34.504 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T02:16:34.504 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T02:16:40.459 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 16.0 MB in 2s (7,476 kB/s) 2024-09-16T02:16:41.908 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-16T02:16:41.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:16:41.993 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-16T02:16:42.206 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-16T02:16:42.206 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-16T02:16:42.394 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:16:42.394 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:16:42.394 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:16:42.394 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:16:42.394 INFO:teuthology.orchestra.run.smithi137.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T02:16:42.395 INFO:teuthology.orchestra.run.smithi137.stdout: python2.7-minimal 2024-09-16T02:16:42.395 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:16:42.396 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2024-09-16T02:16:42.396 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T02:16:42.396 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T02:16:42.397 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T02:16:42.400 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2024-09-16T02:16:42.400 INFO:teuthology.orchestra.run.smithi137.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T02:16:42.400 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2024-09-16T02:16:42.400 INFO:teuthology.orchestra.run.smithi137.stdout: thermald 2024-09-16T02:16:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2024-09-16T02:16:42.450 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T02:16:42.450 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T02:16:42.451 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T02:16:42.452 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2024-09-16T02:16:42.453 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T02:16:42.504 INFO:teuthology.orchestra.run.smithi137.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:16:42.504 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 113 MB of archives. 2024-09-16T02:16:42.504 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T02:16:42.504 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T02:16:42.945 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T02:16:43.055 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T02:16:43.822 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T02:16:43.823 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T02:16:43.823 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T02:16:43.948 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T02:16:44.046 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T02:16:44.483 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 113 MB in 2s (71.4 MB/s) 2024-09-16T02:16:44.701 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T02:16:46.716 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T02:16:46.720 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:16:46.798 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:16:49.316 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T02:16:49.337 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:16:49.424 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:16:49.868 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T02:16:49.889 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:16:49.927 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:16:56.318 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:16:56.419 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:16:56.703 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:16:56.831 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:16:57.159 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T02:16:57.181 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T02:16:57.226 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:17:02.809 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T02:17:02.845 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:17:02.875 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:17:05.566 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:17:05.677 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:17:06.002 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:17:06.124 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:17:06.258 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T02:17:06.378 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:17:07.673 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:17:07.673 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T02:17:07.673 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:17:07.673 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T02:17:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:17:09.265 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:17:10.875 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T02:17:10.976 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T02:17:11.077 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:17:11.127 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T02:17:11.127 INFO:teuthology.orchestra.run.smithi137.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:17:26.176 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T02:17:26.177 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub' 2024-09-16T02:17:26.184 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:17:26.199 INFO:teuthology.orchestra.run.smithi137.stdout:Generating grub configuration file ... 2024-09-16T02:17:26.529 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:17:26.556 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:17:26.872 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:17:26.878 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:17:26.943 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:17:26.948 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:17:27.107 INFO:teuthology.orchestra.run.smithi137.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:17:27.107 INFO:teuthology.orchestra.run.smithi137.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:17:27.107 INFO:teuthology.orchestra.run.smithi137.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:17:27.160 INFO:teuthology.orchestra.run.smithi137.stdout:done 2024-09-16T02:17:27.655 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2024-09-16T02:17:27.715 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2024-09-16T02:17:27.715 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2024-09-16T02:17:27.715 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 21 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.121.121 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:17:27.716 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2024-09-16T02:17:27.717 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2024-09-16T02:17:27.717 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:17:27.717 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2024-09-16T02:17:27.717 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T02:17:27.717 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T02:17:27.717 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T02:17:27.718 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T02:17:27.718 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T02:17:27.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2024-09-16T02:17:27.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-09-16T02:17:28.069 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:17:28.069 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:17:28.070 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:17:28.084 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:17:29.505 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-16T02:17:29.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:17:29.587 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-16T02:17:29.807 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-16T02:17:29.808 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-16T02:17:29.998 INFO:teuthology.orchestra.run.smithi137.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T02:17:29.999 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:17:29.999 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:17:29.999 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:17:29.999 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:17:29.999 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:17:29.999 INFO:teuthology.orchestra.run.smithi137.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:17:30.000 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:17:30.044 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:17:30.046 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2024-09-16T02:17:30.062 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2024-09-16T02:17:30.062 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2024-09-16T02:17:30.062 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2024-09-16T02:17:30.062 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2024-09-16T02:17:30.062 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 21 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.15.0.121.121 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:17:30.063 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2024-09-16T02:17:30.064 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-16T02:17:30.107 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.ylb3YOloiC 2024-09-16T02:17:30.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ylb3YOloiC 2024-09-16T02:17:30.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.ylb3YOloiC 2024-09-16T02:17:30.280 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.ylb3YOloiC is 10KB 2024-09-16T02:17:30.292 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.ylb3YOloiC 2024-09-16T02:17:30.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T02:17:30.350 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:17:30.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T02:17:30.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T02:17:30.413 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T02:17:30.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo update-grub 2024-09-16T02:17:31.723 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub' 2024-09-16T02:17:31.725 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:17:31.727 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-09-16T02:17:31.962 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:17:31.967 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:17:32.316 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:17:32.321 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:17:32.364 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:17:32.370 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:17:32.479 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:17:32.479 INFO:teuthology.orchestra.run.smithi137.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:17:32.479 INFO:teuthology.orchestra.run.smithi137.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:17:32.495 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-09-16T02:17:32.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-09-16T02:18:02.569 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T02:18:02.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-16T02:18:02.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:18:21.117 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-16T02:18:30.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-16T02:18:30.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:18:33.181 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-16T02:18:45.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-16T02:18:45.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:19:16.945 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-16T02:19:17.570 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-16T02:19:17.570 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T02:19:17.571 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-16T02:19:17.617 INFO:teuthology.orchestra.run.smithi137.stdout:5.15.0-121-generic 2024-09-16T02:19:17.617 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T02:19:17.617 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T02:19:17.617 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T02:19:18.619 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T02:19:18.619 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T02:19:18.949 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-09-16T02:19:18.991 DEBUG:teuthology.parallel:result is None 2024-09-16T02:19:18.991 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T02:19:18.997 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T02:19:18.998 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T02:19:19.003 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T02:19:19.010 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T02:19:19.018 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T02:19:19.018 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T02:19:19.068 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T02:19:19.075 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T02:19:19.075 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T02:19:19.129 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:19:19.138 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:19:19.139 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T02:19:19.149 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T02:19:19.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T02:19:19.193 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T02:19:19.299 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T02:19:19.299 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T02:19:19.305 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T02:19:19.353 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T02:19:19.402 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:19:19.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T02:19:19.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-09-16T02:19:19.545 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T02:19:19.553 INFO:teuthology.task.internal:Starting timer... 2024-09-16T02:19:19.553 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T02:19:19.565 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T02:19:19.573 DEBUG:teuthology.task.selinux:Excluding smithi137: OS 'ubuntu' does not support SELinux 2024-09-16T02:19:19.573 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T02:19:19.573 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T02:19:19.573 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T02:19:19.574 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T02:19:19.582 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T02:19:19.595 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T02:19:19.596 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T02:26:48.922 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-09-16T02:26:48.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-16T02:26:48.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:26:49.019 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-16T02:26:49.102 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-16T02:26:49.102 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T02:26:49.116 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T02:26:49.117 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T02:26:49.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Command line: ntpd -gq 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: ---------------------------------------------------- 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: corporation. Support and training for ntp-4 are 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: available at https://www.nwtime.org/support 2024-09-16T02:26:49.173 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: ---------------------------------------------------- 2024-09-16T02:26:49.174 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: proto: precision = 0.063 usec (-24) 2024-09-16T02:26:49.174 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: basedate set to 2022-02-04 2024-09-16T02:26:49.174 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: gps base set to 2022-02-06 (week 2196) 2024-09-16T02:26:49.174 INFO:teuthology.orchestra.run.smithi137.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:26:49.175 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:26:49.175 INFO:teuthology.orchestra.run.smithi137.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:26:49.177 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:26:49.177 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T02:26:49.177 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T02:26:49.177 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T02:26:49.177 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Listen normally on 3 enp3s0f1 172.21.15.137:123 2024-09-16T02:26:49.177 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Listen normally on 4 lo [::1]:123 2024-09-16T02:26:49.178 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8137%5]:123 2024-09-16T02:26:49.178 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:49 ntpd[18459]: Listening on routing socket on fd #22 for interface updates 2024-09-16T02:26:50.173 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:50 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:50.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:50 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:50.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:50 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:50.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:50 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:50.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:50 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:50.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:50 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:52.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:52.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:52.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:52.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:52 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:53.173 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:53 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:53.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:53 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:53.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:53 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:53.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:53 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:53.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:53 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:53.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:53 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:54.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:54.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:54.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:54.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:54.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:54.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:54.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:54.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:54.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:54.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:54.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:54.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:54 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:55.173 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:55.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:55 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:55.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:55.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:55 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:55.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:55.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:55 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:56.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:56.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:56.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:56.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:56.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:56.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:56.174 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:56.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:56.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:26:56.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:26:56.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:26:56.175 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:26:56.175 INFO:teuthology.orchestra.run.smithi137.stdout:16 Sep 02:26:56 ntpd[18459]: ntpd: time slew -0.000996 s 2024-09-16T02:26:56.175 INFO:teuthology.orchestra.run.smithi137.stdout:ntpd: time slew -0.000996s 2024-09-16T02:26:56.176 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T02:26:56.176 INFO:teuthology.orchestra.run.smithi137.stderr:16 Sep 02:26:56 ntpd[18459]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T02:26:56.237 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:26:56.237 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2024-09-16T02:26:56.237 INFO:teuthology.orchestra.run.smithi137.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:26:56.237 INFO:teuthology.orchestra.run.smithi137.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:26:56.238 INFO:teuthology.orchestra.run.smithi137.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:26:56.238 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:26:56.238 INFO:teuthology.run_tasks:Running task install... 2024-09-16T02:26:56.247 DEBUG:teuthology.task.install:project ceph 2024-09-16T02:26:56.247 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-16T02:26:56.247 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-16T02:26:56.247 INFO:teuthology.task.install:Using flavor: default 2024-09-16T02:26:56.253 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T02:26:56.253 INFO:teuthology.task.install:extra packages: [] 2024-09-16T02:26:56.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-key list | grep Ceph 2024-09-16T02:26:56.338 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T02:26:56.369 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T02:26:56.369 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T02:26:56.370 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T02:26:56.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:26:56.525 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-16T02:26:56.525 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:26:56.639 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:26:56.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T02:26:56.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2024-09-16T02:26:56.820 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:26:56.874 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T02:26:56.896 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-16T02:26:56.917 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T02:26:56.939 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-16T02:26:56.945 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:26:56.966 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-16T02:26:57.021 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T02:26:57.378 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:26:57.540 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 157 kB in 1s (225 kB/s) 2024-09-16T02:26:58.576 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-16T02:26:58.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:26:58.655 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2024-09-16T02:26:58.863 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2024-09-16T02:26:58.864 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2024-09-16T02:26:59.063 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:26:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:26:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:26:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:26:59.063 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:26:59.064 INFO:teuthology.orchestra.run.smithi137.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:26:59.064 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:26:59.064 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2024-09-16T02:26:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T02:26:59.065 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T02:26:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T02:26:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T02:26:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T02:26:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T02:26:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T02:26:59.066 INFO:teuthology.orchestra.run.smithi137.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T02:26:59.067 INFO:teuthology.orchestra.run.smithi137.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T02:26:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T02:26:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T02:26:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet zip 2024-09-16T02:26:59.068 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2024-09-16T02:26:59.068 INFO:teuthology.orchestra.run.smithi137.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2024-09-16T02:26:59.069 INFO:teuthology.orchestra.run.smithi137.stdout: btrfs-tools 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T02:26:59.173 INFO:teuthology.orchestra.run.smithi137.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T02:26:59.174 INFO:teuthology.orchestra.run.smithi137.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T02:26:59.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T02:26:59.176 INFO:teuthology.orchestra.run.smithi137.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T02:26:59.231 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T02:26:59.231 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 244 MB of archives. 2024-09-16T02:26:59.231 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-16T02:26:59.231 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T02:26:59.315 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-16T02:26:59.325 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T02:26:59.328 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T02:26:59.344 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T02:26:59.404 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T02:26:59.408 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T02:26:59.421 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T02:26:59.425 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T02:26:59.426 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T02:26:59.427 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T02:26:59.428 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T02:26:59.436 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T02:26:59.438 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T02:26:59.440 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T02:26:59.442 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T02:26:59.443 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T02:26:59.460 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T02:26:59.462 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T02:26:59.464 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T02:26:59.464 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T02:26:59.465 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T02:26:59.465 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T02:26:59.466 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T02:26:59.466 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T02:26:59.477 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T02:26:59.477 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T02:26:59.494 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T02:26:59.494 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T02:26:59.497 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T02:26:59.497 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T02:26:59.498 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T02:26:59.498 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T02:26:59.499 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T02:26:59.500 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T02:26:59.511 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T02:26:59.513 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T02:26:59.528 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T02:26:59.529 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T02:26:59.534 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T02:26:59.534 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T02:26:59.535 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T02:26:59.535 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T02:26:59.536 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T02:26:59.538 INFO:teuthology.orchestra.run.smithi137.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T02:26:59.545 INFO:teuthology.orchestra.run.smithi137.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T02:26:59.547 INFO:teuthology.orchestra.run.smithi137.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T02:26:59.557 INFO:teuthology.orchestra.run.smithi137.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T02:26:59.559 INFO:teuthology.orchestra.run.smithi137.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T02:26:59.560 INFO:teuthology.orchestra.run.smithi137.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T02:26:59.560 INFO:teuthology.orchestra.run.smithi137.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T02:26:59.563 INFO:teuthology.orchestra.run.smithi137.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T02:26:59.615 INFO:teuthology.orchestra.run.smithi137.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T02:26:59.615 INFO:teuthology.orchestra.run.smithi137.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T02:26:59.617 INFO:teuthology.orchestra.run.smithi137.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T02:26:59.617 INFO:teuthology.orchestra.run.smithi137.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T02:26:59.629 INFO:teuthology.orchestra.run.smithi137.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T02:26:59.632 INFO:teuthology.orchestra.run.smithi137.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-16T02:26:59.646 INFO:teuthology.orchestra.run.smithi137.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T02:26:59.648 INFO:teuthology.orchestra.run.smithi137.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T02:26:59.648 INFO:teuthology.orchestra.run.smithi137.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T02:26:59.664 INFO:teuthology.orchestra.run.smithi137.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T02:26:59.664 INFO:teuthology.orchestra.run.smithi137.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T02:26:59.664 INFO:teuthology.orchestra.run.smithi137.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T02:26:59.665 INFO:teuthology.orchestra.run.smithi137.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T02:26:59.666 INFO:teuthology.orchestra.run.smithi137.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T02:26:59.666 INFO:teuthology.orchestra.run.smithi137.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T02:26:59.668 INFO:teuthology.orchestra.run.smithi137.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T02:26:59.668 INFO:teuthology.orchestra.run.smithi137.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T02:26:59.672 INFO:teuthology.orchestra.run.smithi137.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T02:26:59.688 INFO:teuthology.orchestra.run.smithi137.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T02:26:59.688 INFO:teuthology.orchestra.run.smithi137.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T02:26:59.688 INFO:teuthology.orchestra.run.smithi137.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T02:26:59.690 INFO:teuthology.orchestra.run.smithi137.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T02:26:59.692 INFO:teuthology.orchestra.run.smithi137.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T02:26:59.693 INFO:teuthology.orchestra.run.smithi137.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T02:26:59.693 INFO:teuthology.orchestra.run.smithi137.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T02:26:59.702 INFO:teuthology.orchestra.run.smithi137.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T02:26:59.703 INFO:teuthology.orchestra.run.smithi137.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T02:26:59.705 INFO:teuthology.orchestra.run.smithi137.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T02:26:59.719 INFO:teuthology.orchestra.run.smithi137.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T02:26:59.720 INFO:teuthology.orchestra.run.smithi137.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T02:26:59.720 INFO:teuthology.orchestra.run.smithi137.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T02:26:59.722 INFO:teuthology.orchestra.run.smithi137.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T02:26:59.722 INFO:teuthology.orchestra.run.smithi137.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T02:26:59.728 INFO:teuthology.orchestra.run.smithi137.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T02:26:59.729 INFO:teuthology.orchestra.run.smithi137.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T02:26:59.736 INFO:teuthology.orchestra.run.smithi137.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T02:26:59.738 INFO:teuthology.orchestra.run.smithi137.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T02:26:59.739 INFO:teuthology.orchestra.run.smithi137.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T02:26:59.778 INFO:teuthology.orchestra.run.smithi137.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-16T02:26:59.802 INFO:teuthology.orchestra.run.smithi137.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-16T02:26:59.809 INFO:teuthology.orchestra.run.smithi137.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-16T02:26:59.810 INFO:teuthology.orchestra.run.smithi137.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-16T02:26:59.823 INFO:teuthology.orchestra.run.smithi137.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-16T02:26:59.824 INFO:teuthology.orchestra.run.smithi137.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-16T02:26:59.831 INFO:teuthology.orchestra.run.smithi137.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-16T02:27:00.046 INFO:teuthology.orchestra.run.smithi137.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-16T02:27:00.047 INFO:teuthology.orchestra.run.smithi137.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-16T02:27:00.052 INFO:teuthology.orchestra.run.smithi137.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-16T02:27:00.657 INFO:teuthology.orchestra.run.smithi137.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-16T02:27:00.724 INFO:teuthology.orchestra.run.smithi137.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-16T02:27:00.726 INFO:teuthology.orchestra.run.smithi137.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-16T02:27:00.728 INFO:teuthology.orchestra.run.smithi137.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-16T02:27:00.741 INFO:teuthology.orchestra.run.smithi137.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-16T02:27:00.833 INFO:teuthology.orchestra.run.smithi137.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-16T02:27:01.294 INFO:teuthology.orchestra.run.smithi137.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-16T02:27:01.294 INFO:teuthology.orchestra.run.smithi137.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-16T02:27:01.304 INFO:teuthology.orchestra.run.smithi137.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-16T02:27:01.331 INFO:teuthology.orchestra.run.smithi137.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-16T02:27:01.340 INFO:teuthology.orchestra.run.smithi137.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-16T02:27:01.342 INFO:teuthology.orchestra.run.smithi137.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-16T02:27:02.634 INFO:teuthology.orchestra.run.smithi137.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-16T02:27:02.744 INFO:teuthology.orchestra.run.smithi137.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-16T02:27:02.744 INFO:teuthology.orchestra.run.smithi137.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-16T02:27:03.796 INFO:teuthology.orchestra.run.smithi137.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-16T02:27:03.799 INFO:teuthology.orchestra.run.smithi137.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-16T02:27:03.800 INFO:teuthology.orchestra.run.smithi137.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-16T02:27:04.108 INFO:teuthology.orchestra.run.smithi137.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-16T02:27:04.681 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 244 MB in 5s (49.5 MB/s) 2024-09-16T02:27:04.846 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T02:27:04.943 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T02:27:04.949 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T02:27:05.037 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:27:05.340 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T02:27:05.367 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T02:27:05.407 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:27:05.710 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T02:27:05.736 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T02:27:05.793 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:27:06.230 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T02:27:06.257 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:27:06.305 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:27:06.667 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T02:27:06.683 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:27:06.716 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:27:07.061 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T02:27:07.087 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:27:07.136 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:27:07.489 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T02:27:07.515 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T02:27:07.555 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:27:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librados2. 2024-09-16T02:27:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:07.908 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:08.369 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libnbd0. 2024-09-16T02:27:08.396 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T02:27:08.444 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T02:27:08.755 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librbd1. 2024-09-16T02:27:08.781 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:08.822 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:09.233 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T02:27:09.259 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:09.291 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:09.535 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rados. 2024-09-16T02:27:09.561 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:09.627 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:09.930 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T02:27:09.956 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:09.996 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:10.249 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T02:27:10.271 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:10.315 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:10.593 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T02:27:10.619 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:10.651 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:10.946 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T02:27:10.972 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T02:27:11.004 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:27:11.315 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T02:27:11.342 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T02:27:11.382 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T02:27:11.626 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T02:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:11.684 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:12.004 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T02:27:12.030 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T02:27:12.071 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:27:12.306 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T02:27:12.322 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T02:27:12.356 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:27:12.909 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T02:27:12.924 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T02:27:12.967 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:27:13.296 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua5.1. 2024-09-16T02:27:13.323 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T02:27:13.363 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:27:13.649 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-any. 2024-09-16T02:27:13.675 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-16T02:27:13.715 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T02:27:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package zip. 2024-09-16T02:27:13.985 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-16T02:27:14.026 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T02:27:14.346 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package unzip. 2024-09-16T02:27:14.372 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T02:27:14.412 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:27:14.750 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package luarocks. 2024-09-16T02:27:14.776 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T02:27:14.816 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:27:15.149 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librgw2. 2024-09-16T02:27:15.164 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:15.228 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:15.809 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T02:27:15.825 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:15.867 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:16.195 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T02:27:16.222 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T02:27:16.262 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:27:16.565 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T02:27:16.592 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:16.631 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:16.988 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-common. 2024-09-16T02:27:17.012 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:17.043 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:18.665 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-base. 2024-09-16T02:27:18.692 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:18.741 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:19.319 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T02:27:19.345 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T02:27:19.385 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:27:19.688 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T02:27:19.715 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T02:27:19.755 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:27:20.050 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T02:27:20.076 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T02:27:20.116 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:27:20.360 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T02:27:20.383 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T02:27:20.427 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:27:20.689 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T02:27:20.715 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T02:27:20.747 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:27:21.075 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T02:27:21.102 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T02:27:21.141 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T02:27:21.444 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-portend. 2024-09-16T02:27:21.471 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-16T02:27:21.502 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T02:27:21.805 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T02:27:21.831 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T02:27:21.872 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T02:27:22.158 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T02:27:22.184 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T02:27:22.224 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:27:22.586 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T02:27:22.613 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T02:27:22.652 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T02:27:23.149 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T02:27:23.176 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T02:27:23.216 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T02:27:23.519 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T02:27:23.545 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T02:27:23.585 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:27:23.880 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-mako. 2024-09-16T02:27:23.895 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T02:27:23.929 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:27:24.258 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T02:27:24.285 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T02:27:24.324 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:27:24.569 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T02:27:24.584 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T02:27:24.619 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:27:24.913 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webob. 2024-09-16T02:27:24.940 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T02:27:24.979 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:27:25.333 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T02:27:25.360 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T02:27:25.403 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:27:25.694 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T02:27:25.721 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T02:27:25.760 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:27:26.038 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-paste. 2024-09-16T02:27:26.065 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T02:27:26.105 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:27:26.399 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T02:27:26.427 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T02:27:26.486 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:27:26.769 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T02:27:26.796 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T02:27:26.835 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:27:27.097 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T02:27:27.123 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T02:27:27.163 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T02:27:27.425 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T02:27:27.451 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T02:27:27.483 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:27:27.837 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T02:27:27.864 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T02:27:27.904 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T02:27:28.224 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-idna. 2024-09-16T02:27:28.250 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T02:27:28.324 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:27:28.585 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T02:27:28.611 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T02:27:28.651 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:27:28.912 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-requests. 2024-09-16T02:27:28.928 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T02:27:28.971 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:27:29.274 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T02:27:29.301 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T02:27:29.341 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:27:29.627 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T02:27:29.653 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:29.693 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:30.107 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T02:27:30.133 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:30.173 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:30.426 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T02:27:30.441 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T02:27:30.476 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:27:30.778 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T02:27:30.805 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T02:27:30.845 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T02:27:31.198 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T02:27:31.224 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:31.265 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:31.618 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T02:27:31.644 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:31.684 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:33.088 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T02:27:33.116 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:33.154 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:34.258 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph. 2024-09-16T02:27:34.273 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:34.316 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:34.635 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T02:27:34.662 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:34.701 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:35.055 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T02:27:35.082 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:35.113 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:35.516 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package cephadm. 2024-09-16T02:27:35.543 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:35.582 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:35.893 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T02:27:35.921 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T02:27:35.960 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T02:27:36.279 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T02:27:36.295 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:27:36.337 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:27:37.317 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-babel. 2024-09-16T02:27:37.331 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:27:37.375 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:27:37.670 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T02:27:37.697 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T02:27:37.761 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:27:38.106 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T02:27:38.133 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:38.173 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:38.535 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T02:27:38.561 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T02:27:38.601 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T02:27:38.887 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-routes. 2024-09-16T02:27:38.914 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T02:27:38.945 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:27:39.273 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T02:27:39.301 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:39.340 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:42.086 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T02:27:42.103 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T02:27:42.135 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:27:42.598 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T02:27:42.625 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T02:27:42.672 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:27:43.034 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T02:27:43.063 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T02:27:43.101 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:27:43.429 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T02:27:43.456 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T02:27:43.572 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:27:44.439 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T02:27:44.453 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:44.497 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:45.418 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T02:27:45.446 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T02:27:45.484 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T02:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T02:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T02:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:27:46.201 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T02:27:46.228 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T02:27:46.273 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:27:46.576 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T02:27:46.604 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-16T02:27:46.642 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T02:27:46.954 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T02:27:46.982 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T02:27:47.021 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T02:27:47.341 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T02:27:47.356 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T02:27:47.399 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:27:47.685 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T02:27:47.713 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T02:27:47.751 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T02:27:48.071 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T02:27:48.099 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T02:27:48.138 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:27:48.810 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T02:27:48.838 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:48.877 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:49.188 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T02:27:49.215 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T02:27:49.254 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:27:49.566 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T02:27:49.593 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:27:49.632 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:27:49.885 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package jq. 2024-09-16T02:27:49.901 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:27:49.943 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T02:27:50.195 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package socat. 2024-09-16T02:27:50.211 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T02:27:50.253 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:27:50.540 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T02:27:50.556 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T02:27:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:27:50.944 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-test. 2024-09-16T02:27:50.960 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:50.994 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:53.149 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T02:27:53.176 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:27:53.214 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:53.494 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T02:27:53.521 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:27:53.560 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:27:54.021 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T02:27:54.049 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T02:27:54.087 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:27:54.417 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T02:27:54.445 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T02:27:54.484 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:27:54.829 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T02:27:54.857 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T02:27:54.920 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:27:55.334 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package pkg-config. 2024-09-16T02:27:55.362 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T02:27:55.401 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:27:55.662 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T02:27:55.678 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T02:27:55.720 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:27:56.131 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T02:27:56.159 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T02:27:56.198 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T02:27:56.501 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T02:27:56.528 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T02:27:56.567 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T02:27:56.871 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T02:27:56.898 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T02:27:56.954 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:27:57.240 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T02:27:57.268 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T02:27:57.315 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T02:27:57.651 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-py. 2024-09-16T02:27:57.679 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-16T02:27:57.742 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T02:27:58.021 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T02:27:58.049 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T02:27:58.087 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:27:58.583 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T02:27:58.611 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T02:27:58.649 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:27:58.961 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-toml. 2024-09-16T02:27:58.989 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-16T02:27:59.027 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T02:27:59.304 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T02:27:59.321 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T02:27:59.354 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T02:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T02:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:28:00.319 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T02:28:00.336 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T02:28:00.377 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:28:01.108 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package radosgw. 2024-09-16T02:28:01.134 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:28:01.172 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:02.075 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T02:28:02.092 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:28:02.136 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:02.430 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T02:28:02.672 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:28:02.773 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:28:03.045 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T02:28:03.362 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T02:28:03.796 INFO:teuthology.orchestra.run.smithi137.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T02:28:03.885 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:04.469 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user cephadm....done 2024-09-16T02:28:04.626 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:28:05.269 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:28:05.520 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:28:05.647 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:28:05.898 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T02:28:06.182 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:28:06.308 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T02:28:06.619 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:28:06.970 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T02:28:07.221 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T02:28:07.481 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:28:07.724 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:28:07.975 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:28:08.376 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:28:08.627 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:08.903 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:28:09.056 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T02:28:09.180 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:28:09.306 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:28:09.432 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:28:09.683 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:28:10.087 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T02:28:10.122 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T02:28:10.195 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:28:10.313 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T02:28:10.655 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:28:10.781 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T02:28:11.049 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:28:11.317 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:28:11.635 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T02:28:11.760 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:28:12.446 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:28:12.697 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:28:12.798 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:28:12.924 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:28:13.216 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T02:28:13.609 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T02:28:13.852 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:28:14.212 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:28:14.505 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:28:14.848 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:28:15.099 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:28:15.533 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:15.843 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:28:16.144 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:28:17.349 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:28:17.618 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:28:17.735 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T02:28:17.987 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:28:18.096 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:28:18.197 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:28:18.473 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:28:18.724 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T02:28:18.983 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:28:19.101 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:28:19.403 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:28:19.654 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T02:28:19.897 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T02:28:20.148 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:28:20.423 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T02:28:20.700 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:28:20.826 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T02:28:21.102 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:28:21.219 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:28:21.495 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:28:21.772 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:28:22.023 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:28:22.149 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:28:22.275 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:28:22.364 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T02:28:22.610 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:28:22.753 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:28:23.137 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:28:23.388 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T02:28:23.514 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T02:28:23.773 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:28:23.908 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:28:24.217 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:28:24.519 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:28:24.786 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T02:28:24.887 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T02:28:25.138 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T02:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:28:25.751 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:28:26.128 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T02:28:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:28:26.723 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:28:26.849 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:26.975 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:27.105 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:28:27.404 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:27.505 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:27.623 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T02:28:27.899 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:28.034 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:28.152 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:28.355 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:28:28.355 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:28:28.811 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:28.912 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:29.013 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:29.122 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:28:30.309 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:30.419 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:30.528 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:30.654 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:30.788 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:31.307 INFO:teuthology.orchestra.run.smithi137.stdout:Adding group ceph....done 2024-09-16T02:28:31.658 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user ceph....done 2024-09-16T02:28:31.827 INFO:teuthology.orchestra.run.smithi137.stdout:Setting system user ceph properties....done 2024-09-16T02:28:31.841 INFO:teuthology.orchestra.run.smithi137.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T02:28:31.975 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T02:28:32.295 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T02:28:32.756 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:32.882 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:33.349 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:28:33.349 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:28:33.826 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:34.153 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T02:28:34.627 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:28:35.304 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:35.507 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:28:35.507 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:28:35.972 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:36.253 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:28:36.253 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:28:36.715 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:36.842 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:36.993 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:37.202 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:28:37.202 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:28:37.644 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:37.803 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:37.929 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:38.080 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:28:38.426 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T02:28:38.534 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T02:28:38.739 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T02:28:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T02:28:40.562 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T02:28:40.568 DEBUG:teuthology.parallel:result is None 2024-09-16T02:28:40.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:28:40.709 DEBUG:teuthology.orchestra.run.smithi137:> dpkg-query -W -f '${Version}' ceph 2024-09-16T02:28:40.728 INFO:teuthology.orchestra.run.smithi137.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:28:40.728 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:28:40.728 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-16T02:28:40.730 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T02:28:40.731 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:40.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T02:28:40.785 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T02:28:40.785 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:40.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T02:28:40.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T02:28:40.895 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T02:28:40.895 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:40.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T02:28:40.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T02:28:41.007 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T02:28:41.007 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:41.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T02:28:41.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T02:28:41.119 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T02:28:41.204 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T02:28:41.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-09-16T02:28:41.220 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T02:28:41.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T02:28:41.275 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T02:28:41.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T02:28:41.338 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T02:28:41.339 INFO:tasks.ceph:config {'conf': {'osd': {'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T02:28:41.339 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763', 'branch': 'main', 'description': 'rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905763', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 2855, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'sleep_before_teardown': 0, 'subset': '17936/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7498493279982dfe87f00616198a5967475f6169', 'targets': {'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPJblbmhzcwjMyiLmZgPA0g84vuWylaB01uL+2pFBi9nEd68oVqGMf85pcUCipYyfnUxn5gsxLZDthiokvRr0Tg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'fs': 'xfs', 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'cluster': 'ceph'}}, {'osd_recovery.test_incomplete_pgs': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_20:00:16', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T02:28:41.339 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T02:28:41.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T02:28:41.436 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:41.436 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T02:28:41.484 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T02:28:41.484 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-16 02:23:17.957833921 +0000 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-16 02:23:17.953833957 +0000 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-16 02:23:17.953833957 +0000 2024-09-16T02:28:41.533 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-16T02:28:41.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T02:28:41.589 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-16T02:28:41.590 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-16T02:28:41.590 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000355801 s, 1.4 MB/s 2024-09-16T02:28:41.590 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T02:28:41.637 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-09-16T02:28:41.684 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T02:28:41.684 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:28:41.684 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-09-16T02:28:41.684 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:28:41.685 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-16 02:23:18.733826990 +0000 2024-09-16T02:28:41.685 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-16 02:23:18.489829170 +0000 2024-09-16T02:28:41.685 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-16 02:23:18.489829170 +0000 2024-09-16T02:28:41.685 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-16T02:28:41.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T02:28:41.738 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-16T02:28:41.738 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-16T02:28:41.738 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000323617 s, 1.6 MB/s 2024-09-16T02:28:41.739 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T02:28:41.789 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-16 02:23:19.217822667 +0000 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-16 02:23:18.985824740 +0000 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-16 02:23:18.985824740 +0000 2024-09-16T02:28:41.836 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-16T02:28:41.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T02:28:41.890 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-16T02:28:41.890 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-16T02:28:41.890 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000443031 s, 1.2 MB/s 2024-09-16T02:28:41.892 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T02:28:41.941 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-09-16T02:28:41.987 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T02:28:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:28:41.988 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-09-16T02:28:41.988 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:28:41.988 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-16 02:23:19.801817451 +0000 2024-09-16T02:28:41.988 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-16 02:23:19.481820309 +0000 2024-09-16T02:28:41.988 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-16 02:23:19.481820309 +0000 2024-09-16T02:28:41.988 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-09-16T02:28:41.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T02:28:42.042 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-16T02:28:42.042 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-16T02:28:42.042 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000389897 s, 1.3 MB/s 2024-09-16T02:28:42.043 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T02:28:42.093 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T02:28:42.093 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-16T02:28:42.093 INFO:tasks.ceph:Generating config... 2024-09-16T02:28:42.094 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2024-09-16T02:28:42.094 INFO:tasks.ceph:[osd] osd_fast_fail_on_connection_refused = False 2024-09-16T02:28:42.094 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T02:28:42.094 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] bluestore write v2 = True 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T02:28:42.095 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T02:28:42.096 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T02:28:42.096 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T02:28:42.096 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T02:28:42.096 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-16T02:28:42.096 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T02:28:42.096 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T02:28:42.096 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T02:28:42.096 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T02:28:42.096 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T02:28:42.096 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T02:28:42.096 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T02:28:42.096 INFO:tasks.ceph:[global] ms type = async 2024-09-16T02:28:42.097 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T02:28:42.097 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T02:28:42.097 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T02:28:42.097 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T02:28:42.097 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T02:28:42.097 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T02:28:42.097 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T02:28:42.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T02:28:42.168 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T02:28:42.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T02:28:42.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T02:28:42.315 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.137:6789]'), ('mon.b', '[v1:172.21.15.137:6790]'), ('mon.c', '[v1:172.21.15.137:6791]')] 2024-09-16T02:28:42.315 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.137:6789],[v1:172.21.15.137:6790],[v1:172.21.15.137:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T02:28:42.316 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:42.316 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T02:28:42.364 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.137:6789]' --addv b '[v1:172.21.15.137:6790]' --addv c '[v1:172.21.15.137:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:28:42.449 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:28:42.449 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: generated fsid eabc9450-e776-4be8-8adb-00ce348fa2c2 2024-09-16T02:28:42.449 INFO:teuthology.orchestra.run.smithi137.stdout:setting min_mon_release = quincy 2024-09-16T02:28:42.449 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 0 2024-09-16T02:28:42.449 INFO:teuthology.orchestra.run.smithi137.stdout:fsid eabc9450-e776-4be8-8adb-00ce348fa2c2 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:last_changed 2024-09-16T02:28:42.448786+0000 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-09-16T02:28:42.448786+0000 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:min_mon_release 17 (quincy) 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:election_strategy: 1 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:0: v1:172.21.15.137:6789/0 mon.a 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:1: v1:172.21.15.137:6790/0 mon.b 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:2: v1:172.21.15.137:6791/0 mon.c 2024-09-16T02:28:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T02:28:42.451 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T02:28:42.500 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID eabc9450-e776-4be8-8adb-00ce348fa2c2... 2024-09-16T02:28:42.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T02:28:42.574 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-09-16T02:28:42.574 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-09-16T02:28:42.574 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T02:28:42.575 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-09-16T02:28:42.576 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-09-16T02:28:42.577 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = "[v1:172.21.15.137:6789],[v1:172.21.15.137:6790],[v1:172.21.15.137:6791]" 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon client directed command retry = 5 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon election default strategy = 1 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: ms bind msgr2 = False 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: ms inject socket failures = 5000 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: ms type = async 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = eabc9450-e776-4be8-8adb-00ce348fa2c2 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-09-16T02:28:42.578 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug trim objects = true 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-09-16T02:28:42.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd min pg log entries = 5 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd_fast_fail_on_connection_refused = False 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: bdev async discard = True 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: bdev enable discard = True 2024-09-16T02:28:42.580 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore block size = 96636764160 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore fsck on mount = True 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore write v2 = True 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore zero block detection = True 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluefs = 20 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluestore = 20 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: debug rocksdb = 10 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd full ratio = 0.9 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd nearfull ratio = 0.8 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify cached snaps = True 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify missing on start = True 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd failsafe full ratio = 0.95 2024-09-16T02:28:42.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock override recovery settings = True 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: osd memory target = 1610612736 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = bluestore 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: osd op queue = debug_random 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: osd op queue cut off = debug_random 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.582 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-09-16T02:28:42.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout: mon scrub interval = 300 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-09-16T02:28:42.585 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.b] 2024-09-16T02:28:42.586 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.c] 2024-09-16T02:28:42.586 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T02:28:42.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T02:28:42.675 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T02:28:42.675 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:42.675 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T02:28:42.681 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:42.681 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T02:28:42.727 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:28:42.728 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:42.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T02:28:42.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T02:28:42.792 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:42.792 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:28:42.840 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T02:28:42.840 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T02:28:42.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T02:28:42.924 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T02:28:42.927 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T02:28:42.927 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T02:28:42.927 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T02:28:42.927 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-16T02:28:42.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T02:28:42.982 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T02:28:42.982 INFO:tasks.ceph:role: osd.0 2024-09-16T02:28:42.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:28:42.983 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T02:28:43.046 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:28:43.046 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:28:43.046 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:28:43.046 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:28:43.046 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:28:43.047 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:28:43.047 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:28:43.047 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:28:43.047 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:28:43.047 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:28:43.500 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-09-16T02:28:43.502 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-09-16T02:28:43.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T02:28:43.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T02:28:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:28:43.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:28:43.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T02:28:43.653 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T02:28:43.653 INFO:tasks.ceph:role: osd.1 2024-09-16T02:28:43.653 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:28:43.653 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T02:28:43.714 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:28:43.715 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:28:44.187 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-09-16T02:28:44.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-09-16T02:28:44.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T02:28:44.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T02:28:44.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:28:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:28:44.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T02:28:44.411 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T02:28:44.412 INFO:tasks.ceph:role: osd.2 2024-09-16T02:28:44.412 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:28:44.412 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:28:44.476 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:28:44.943 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-09-16T02:28:44.945 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-09-16T02:28:44.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T02:28:45.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T02:28:45.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:28:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:28:45.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T02:28:45.167 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T02:28:45.168 INFO:tasks.ceph:role: osd.3 2024-09-16T02:28:45.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:28:45.168 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T02:28:45.231 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:28:45.231 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:28:45.232 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:28:45.657 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-09-16T02:28:45.660 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-09-16T02:28:45.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T02:28:45.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T02:28:45.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:28:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:28:45.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:28:45.872 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:45.867+0000 7f1eecc9e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T02:28:45.874 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:45.867+0000 7f1eecc9e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T02:28:45.874 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:45.867+0000 7f1eecc9e7c0 -1 bdev(0x561f803d9800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T02:28:45.874 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:45.867+0000 7f1eecc9e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T02:28:49.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T02:28:49.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:28:49.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:49.087+0000 7f2f1f2387c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T02:28:49.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:49.087+0000 7f2f1f2387c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T02:28:49.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:49.087+0000 7f2f1f2387c0 -1 bdev(0x56328a7ff800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T02:28:49.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:49.087+0000 7f2f1f2387c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T02:28:52.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T02:28:52.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:28:52.365 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:52.359+0000 7fd3fbec77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T02:28:52.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:52.359+0000 7fd3fbec77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T02:28:52.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:52.359+0000 7fd3fbec77c0 -1 bdev(0x55d909d97800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T02:28:52.366 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:52.359+0000 7fd3fbec77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T02:28:55.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T02:28:55.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:28:55.635 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:55.631+0000 7f8ff9c827c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T02:28:55.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:55.631+0000 7f8ff9c827c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T02:28:55.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:55.631+0000 7f8ff9c827c0 -1 bdev(0x55d6f6c37800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T02:28:55.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:28:55.631+0000 7f8ff9c827c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T02:28:58.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T02:28:58.816 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T02:28:58.816 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:58.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T02:28:58.871 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:58.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T02:28:58.927 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:58.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T02:28:58.983 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:58.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T02:28:59.040 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:28:59.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T02:28:59.094 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T02:28:59.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T02:28:59.149 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-09-16T02:28:59.149 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDal+dmEHAaNxAAZ7r2EA2jlTf8jmMsefxi6g== 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDdl+dm4rLzMxAAtDqDHaGQsj9EW119u4D5vA== 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDhl+dmjSiFBRAAWMNG5f5R0DPbJDeNsNfA7Q== 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDkl+dmiEfLFRAAqbguTe8+iQH7BrwSOEdRzw== 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.3] 2024-09-16T02:28:59.150 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDnl+dmnojgJRAAAzfrZbq40HOIgbcyv0PM2g== 2024-09-16T02:28:59.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T02:28:59.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:28:59.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:28:59.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:28:59.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:28:59.497 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T02:28:59.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T02:28:59.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T02:28:59.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T02:28:59.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T02:28:59.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T02:28:59.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T02:28:59.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T02:28:59.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T02:28:59.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T02:28:59.998 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:29:00.043 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T02:29:00.043 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T02:29:00.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T02:29:00.088 INFO:tasks.ceph.mon.a:Started 2024-09-16T02:29:00.089 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T02:29:00.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T02:29:00.091 INFO:tasks.ceph.mon.b:Started 2024-09-16T02:29:00.091 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T02:29:00.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T02:29:00.093 INFO:tasks.ceph.mon.c:Started 2024-09-16T02:29:00.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph config set mgr mgr_pool false --force 2024-09-16T02:29:00.533 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T02:29:00.533 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T02:29:00.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T02:29:00.535 INFO:tasks.ceph.mgr.x:Started 2024-09-16T02:29:00.535 DEBUG:tasks.ceph:set 0 configs 2024-09-16T02:29:00.536 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T02:29:00.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T02:29:00.737 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-09-16T02:29:00.755 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T02:29:00.755 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T02:29:00.755 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:29:00.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T02:29:00.766 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:29:00.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T02:29:00.816 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:29:00.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T02:29:00.869 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-16T02:29:00.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T02:29:00.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 12c8d8a5-d99d-4bfc-99dd-7b19e351ed44 0 2024-09-16T02:29:01.159 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-09-16T02:29:01.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 8a607138-b9f7-49d3-941d-ca065ebe247f 1 2024-09-16T02:29:01.371 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-09-16T02:29:01.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82 2 2024-09-16T02:29:01.585 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-09-16T02:29:01.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a 3 2024-09-16T02:29:01.788 INFO:teuthology.orchestra.run.smithi137.stdout:3 2024-09-16T02:29:01.804 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T02:29:01.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T02:29:01.806 INFO:tasks.ceph.osd.0:Started 2024-09-16T02:29:01.806 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T02:29:01.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T02:29:01.808 INFO:tasks.ceph.osd.1:Started 2024-09-16T02:29:01.809 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T02:29:01.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T02:29:01.811 INFO:tasks.ceph.osd.2:Started 2024-09-16T02:29:01.811 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T02:29:01.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T02:29:01.813 INFO:tasks.ceph.osd.3:Started 2024-09-16T02:29:01.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:29:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:02.012 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":6,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:29:01.784965+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T02:29:01.784965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:29:02.028 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T02:29:02.028 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T02:29:02.430 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T02:29:02.607 INFO:teuthology.misc.health.smithi137.stdout: 2024-09-16T02:29:02.607 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":6,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:29:01.784965+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T02:29:01.784965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:29:02.621 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-16T02:29:02.932 INFO:tasks.ceph.osd.3.smithi137.stderr:2024-09-16T02:29:02.927+0000 7f37e977d7c0 -1 Falling back to public interface 2024-09-16T02:29:02.988 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-16T02:29:02.983+0000 7f2722b397c0 -1 Falling back to public interface 2024-09-16T02:29:03.461 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-16T02:29:03.455+0000 7f65eb1f67c0 -1 Falling back to public interface 2024-09-16T02:29:03.472 INFO:tasks.ceph.mgr.x.smithi137.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T02:29:03.473 INFO:tasks.ceph.mgr.x.smithi137.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T02:29:03.473 INFO:tasks.ceph.mgr.x.smithi137.stderr: from numpy import show_config as show_numpy_config 2024-09-16T02:29:03.700 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-16T02:29:03.695+0000 7fd5089337c0 -1 Falling back to public interface 2024-09-16T02:29:06.403 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-16T02:29:06.395+0000 7f7e0b0e8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T02:29:06.404 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-09-16T02:29:06.399+0000 7f7e0b0e8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T02:29:07.172 INFO:tasks.ceph.osd.3.smithi137.stderr:2024-09-16T02:29:07.167+0000 7f37e977d7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T02:29:07.470 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-16T02:29:07.463+0000 7f2722b397c0 -1 osd.0 0 log_to_monitors true 2024-09-16T02:29:07.471 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-16T02:29:07.467+0000 7f65eb1f67c0 -1 osd.2 0 log_to_monitors true 2024-09-16T02:29:07.729 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-16T02:29:07.723+0000 7fd5089337c0 -1 osd.1 0 log_to_monitors true 2024-09-16T02:29:08.468 INFO:tasks.ceph.osd.3.smithi137.stderr:2024-09-16T02:29:08.463+0000 7f37e5726640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T02:29:08.477 INFO:tasks.ceph.osd.3.smithi137.stderr:2024-09-16T02:29:08.471+0000 7f37e1560640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:29:09.025 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T02:29:09.330 INFO:teuthology.misc.health.smithi137.stdout: 2024-09-16T02:29:09.330 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":8,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:29:08.454719+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T02:29:01.784965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:29:09.348 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-16T02:29:09.475 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-16T02:29:09.471+0000 7f271eae2640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T02:29:09.581 INFO:tasks.ceph.osd.0.smithi137.stderr:2024-09-16T02:29:09.575+0000 7f271a91c640 -1 osd.0 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:29:10.707 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-16T02:29:10.703+0000 7f65e79b1640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T02:29:10.713 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-16T02:29:10.707+0000 7f65defd1640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:29:10.728 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-16T02:29:10.723+0000 7fd5050ee640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T02:29:10.733 INFO:tasks.ceph.osd.1.smithi137.stderr:2024-09-16T02:29:10.727+0000 7fd4fc70e640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:29:15.751 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T02:29:16.049 INFO:teuthology.misc.health.smithi137.stdout: 2024-09-16T02:29:16.050 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":12,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:29:12.739706+0000","last_up_change":"2024-09-16T02:29:11.729743+0000","last_in_change":"2024-09-16T02:29:01.784965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":683366439}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":683366439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":683366439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":683366439}]},"public_addr":"172.21.15.137:6804/683366439","cluster_addr":"172.21.15.137:6805/683366439","heartbeat_back_addr":"172.21.15.137:6807/683366439","heartbeat_front_addr":"172.21.15.137:6806/683366439","state":["exists","up"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":4172164026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":4172164026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":4172164026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":4172164026}]},"public_addr":"172.21.15.137:6812/4172164026","cluster_addr":"172.21.15.137:6813/4172164026","heartbeat_back_addr":"172.21.15.137:6815/4172164026","heartbeat_front_addr":"172.21.15.137:6814/4172164026","state":["exists","up"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":2156154695}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":2156154695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":2156154695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":2156154695}]},"public_addr":"172.21.15.137:6808/2156154695","cluster_addr":"172.21.15.137:6809/2156154695","heartbeat_back_addr":"172.21.15.137:6811/2156154695","heartbeat_front_addr":"172.21.15.137:6810/2156154695","state":["exists","up"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6800","nonce":1497967517}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":1497967517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":1497967517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":1497967517}]},"public_addr":"172.21.15.137:6800/1497967517","cluster_addr":"172.21.15.137:6801/1497967517","heartbeat_back_addr":"172.21.15.137:6803/1497967517","heartbeat_front_addr":"172.21.15.137:6802/1497967517","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.492963+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.693906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.471476+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.155330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:29:16.065 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-16T02:29:16.065 INFO:tasks.ceph:Creating RBD pool 2024-09-16T02:29:16.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T02:29:16.464 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'rbd' created 2024-09-16T02:29:16.486 DEBUG:teuthology.orchestra.run.smithi137:> rbd --cluster ceph pool init rbd 2024-09-16T02:29:19.530 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T02:29:19.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T02:29:19.531 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T02:29:19.847 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:19.865 INFO:teuthology.orchestra.run.smithi137.stdout:[{"version":4,"timestamp":"2024-09-16T02:29:10.728808+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"6523.398484"}]}] 2024-09-16T02:29:19.866 INFO:tasks.ceph_manager:config epoch is 4 2024-09-16T02:29:19.866 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T02:29:19.866 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T02:29:19.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T02:29:20.207 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:20.223 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":4,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":2278047882}]},"active_addr":"172.21.15.137:6816/2278047882","active_change":"2024-09-16T02:29:05.385636+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":320038680}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1604936723}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1218913181}]}]} 2024-09-16T02:29:20.227 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T02:29:20.227 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T02:29:20.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:29:20.531 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:20.531 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":16,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:29:19.474161+0000","last_up_change":"2024-09-16T02:29:11.729743+0000","last_in_change":"2024-09-16T02:29:01.784965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T02:29:16.348818+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":683366439}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":683366439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":683366439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":683366439}]},"public_addr":"172.21.15.137:6804/683366439","cluster_addr":"172.21.15.137:6805/683366439","heartbeat_back_addr":"172.21.15.137:6807/683366439","heartbeat_front_addr":"172.21.15.137:6806/683366439","state":["exists","up"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":4172164026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":4172164026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":4172164026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":4172164026}]},"public_addr":"172.21.15.137:6812/4172164026","cluster_addr":"172.21.15.137:6813/4172164026","heartbeat_back_addr":"172.21.15.137:6815/4172164026","heartbeat_front_addr":"172.21.15.137:6814/4172164026","state":["exists","up"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":2156154695}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":2156154695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":2156154695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":2156154695}]},"public_addr":"172.21.15.137:6808/2156154695","cluster_addr":"172.21.15.137:6809/2156154695","heartbeat_back_addr":"172.21.15.137:6811/2156154695","heartbeat_front_addr":"172.21.15.137:6810/2156154695","state":["exists","up"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6800","nonce":1497967517}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":1497967517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":1497967517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":1497967517}]},"public_addr":"172.21.15.137:6800/1497967517","cluster_addr":"172.21.15.137:6801/1497967517","heartbeat_back_addr":"172.21.15.137:6803/1497967517","heartbeat_front_addr":"172.21.15.137:6802/1497967517","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.492963+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.693906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.471476+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.155330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:29:20.549 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T02:29:20.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:29:20.832 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:20.832 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":16,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:29:19.474161+0000","last_up_change":"2024-09-16T02:29:11.729743+0000","last_in_change":"2024-09-16T02:29:01.784965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T02:29:16.348818+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":683366439}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":683366439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":683366439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":683366439}]},"public_addr":"172.21.15.137:6804/683366439","cluster_addr":"172.21.15.137:6805/683366439","heartbeat_back_addr":"172.21.15.137:6807/683366439","heartbeat_front_addr":"172.21.15.137:6806/683366439","state":["exists","up"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":4172164026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":4172164026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":4172164026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":4172164026}]},"public_addr":"172.21.15.137:6812/4172164026","cluster_addr":"172.21.15.137:6813/4172164026","heartbeat_back_addr":"172.21.15.137:6815/4172164026","heartbeat_front_addr":"172.21.15.137:6814/4172164026","state":["exists","up"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":2156154695}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":2156154695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":2156154695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":2156154695}]},"public_addr":"172.21.15.137:6808/2156154695","cluster_addr":"172.21.15.137:6809/2156154695","heartbeat_back_addr":"172.21.15.137:6811/2156154695","heartbeat_front_addr":"172.21.15.137:6810/2156154695","state":["exists","up"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6800","nonce":1497967517}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":1497967517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":1497967517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":1497967517}]},"public_addr":"172.21.15.137:6800/1497967517","cluster_addr":"172.21.15.137:6801/1497967517","heartbeat_back_addr":"172.21.15.137:6803/1497967517","heartbeat_front_addr":"172.21.15.137:6802/1497967517","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.492963+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.693906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.471476+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.155330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:29:20.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T02:29:20.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T02:29:20.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T02:29:20.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T02:29:21.006 INFO:teuthology.orchestra.run.smithi137.stdout:38654705670 2024-09-16T02:29:21.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:29:21.007 INFO:teuthology.orchestra.run.smithi137.stdout:47244640261 2024-09-16T02:29:21.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:29:21.019 INFO:teuthology.orchestra.run.smithi137.stdout:47244640260 2024-09-16T02:29:21.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:29:21.021 INFO:teuthology.orchestra.run.smithi137.stdout:42949672964 2024-09-16T02:29:21.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:29:21.313 INFO:teuthology.orchestra.run.smithi137.stdout:38654705668 2024-09-16T02:29:21.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705668 for osd.3 2024-09-16T02:29:21.349 INFO:teuthology.orchestra.run.smithi137.stdout:47244640260 2024-09-16T02:29:21.350 INFO:teuthology.orchestra.run.smithi137.stdout:42949672963 2024-09-16T02:29:21.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672964 got 42949672963 for osd.0 2024-09-16T02:29:21.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.2 2024-09-16T02:29:21.388 INFO:teuthology.orchestra.run.smithi137.stdout:47244640259 2024-09-16T02:29:21.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.1 2024-09-16T02:29:22.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:29:22.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:29:22.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:29:22.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:29:22.650 INFO:teuthology.orchestra.run.smithi137.stdout:38654705670 2024-09-16T02:29:22.667 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.3 2024-09-16T02:29:22.667 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:22.684 INFO:teuthology.orchestra.run.smithi137.stdout:42949672964 2024-09-16T02:29:22.694 INFO:teuthology.orchestra.run.smithi137.stdout:47244640261 2024-09-16T02:29:22.702 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672964 got 42949672964 for osd.0 2024-09-16T02:29:22.702 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:22.712 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-16T02:29:22.712 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:22.723 INFO:teuthology.orchestra.run.smithi137.stdout:47244640260 2024-09-16T02:29:22.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-09-16T02:29:22.740 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:22.740 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T02:29:22.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:23.065 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:23.065 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:23.081 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T02:29:21.396615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108648,"kb_used_data":552,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378712,"statfs":{"total":386547056640,"available":386435801088,"internally_reserved":0,"allocated":565248,"data_stored":139806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494006+0000","last_change":"2024-09-16T02:29:19.494636+0000","last_active":"2024-09-16T02:29:19.494006+0000","last_peered":"2024-09-16T02:29:19.494006+0000","last_clean":"2024-09-16T02:29:19.494006+0000","last_became_active":"2024-09-16T02:29:17.477993+0000","last_became_peered":"2024-09-16T02:29:17.477993+0000","last_unstale":"2024-09-16T02:29:19.494006+0000","last_undegraded":"2024-09-16T02:29:19.494006+0000","last_fullsized":"2024-09-16T02:29:19.494006+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:05.906602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015503139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494428+0000","last_change":"2024-09-16T02:29:19.494428+0000","last_active":"2024-09-16T02:29:19.494428+0000","last_peered":"2024-09-16T02:29:19.494428+0000","last_clean":"2024-09-16T02:29:19.494428+0000","last_became_active":"2024-09-16T02:29:17.482149+0000","last_became_peered":"2024-09-16T02:29:17.482149+0000","last_unstale":"2024-09-16T02:29:19.494428+0000","last_undegraded":"2024-09-16T02:29:19.494428+0000","last_fullsized":"2024-09-16T02:29:19.494428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:18.897041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.776174+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:19.776174+0000","last_peered":"2024-09-16T02:29:19.776174+0000","last_clean":"2024-09-16T02:29:19.776174+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:19.776174+0000","last_undegraded":"2024-09-16T02:29:19.776174+0000","last_fullsized":"2024-09-16T02:29:19.776174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493583+0000","last_change":"2024-09-16T02:29:19.493583+0000","last_active":"2024-09-16T02:29:19.493583+0000","last_peered":"2024-09-16T02:29:19.493583+0000","last_clean":"2024-09-16T02:29:19.493583+0000","last_became_active":"2024-09-16T02:29:17.482365+0000","last_became_peered":"2024-09-16T02:29:17.482365+0000","last_unstale":"2024-09-16T02:29:19.493583+0000","last_undegraded":"2024-09-16T02:29:19.493583+0000","last_fullsized":"2024-09-16T02:29:19.493583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:02.633320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493864+0000","last_change":"2024-09-16T02:29:19.494610+0000","last_active":"2024-09-16T02:29:19.493864+0000","last_peered":"2024-09-16T02:29:19.493864+0000","last_clean":"2024-09-16T02:29:19.493864+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:19.493864+0000","last_undegraded":"2024-09-16T02:29:19.493864+0000","last_fullsized":"2024-09-16T02:29:19.493864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:13:26.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020900630000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.506253+0000","last_change":"2024-09-16T02:29:19.506253+0000","last_active":"2024-09-16T02:29:19.506253+0000","last_peered":"2024-09-16T02:29:19.506253+0000","last_clean":"2024-09-16T02:29:19.506253+0000","last_became_active":"2024-09-16T02:29:17.486477+0000","last_became_peered":"2024-09-16T02:29:17.486477+0000","last_unstale":"2024-09-16T02:29:19.506253+0000","last_undegraded":"2024-09-16T02:29:19.506253+0000","last_fullsized":"2024-09-16T02:29:19.506253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:48:11.615610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493711+0000","last_change":"2024-09-16T02:29:19.493711+0000","last_active":"2024-09-16T02:29:19.493711+0000","last_peered":"2024-09-16T02:29:19.493711+0000","last_clean":"2024-09-16T02:29:19.493711+0000","last_became_active":"2024-09-16T02:29:17.482292+0000","last_became_peered":"2024-09-16T02:29:17.482292+0000","last_unstale":"2024-09-16T02:29:19.493711+0000","last_undegraded":"2024-09-16T02:29:19.493711+0000","last_fullsized":"2024-09-16T02:29:19.493711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:59.836546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.492730+0000","last_change":"2024-09-16T02:29:19.493362+0000","last_active":"2024-09-16T02:29:19.492730+0000","last_peered":"2024-09-16T02:29:19.492730+0000","last_clean":"2024-09-16T02:29:19.492730+0000","last_became_active":"2024-09-16T02:29:17.482879+0000","last_became_peered":"2024-09-16T02:29:17.482879+0000","last_unstale":"2024-09-16T02:29:19.492730+0000","last_undegraded":"2024-09-16T02:29:19.492730+0000","last_fullsized":"2024-09-16T02:29:19.492730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:38.603073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012747839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:23.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:23.353 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:23.353 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:23.369 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T02:29:21.396615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108648,"kb_used_data":552,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378712,"statfs":{"total":386547056640,"available":386435801088,"internally_reserved":0,"allocated":565248,"data_stored":139806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494006+0000","last_change":"2024-09-16T02:29:19.494636+0000","last_active":"2024-09-16T02:29:19.494006+0000","last_peered":"2024-09-16T02:29:19.494006+0000","last_clean":"2024-09-16T02:29:19.494006+0000","last_became_active":"2024-09-16T02:29:17.477993+0000","last_became_peered":"2024-09-16T02:29:17.477993+0000","last_unstale":"2024-09-16T02:29:19.494006+0000","last_undegraded":"2024-09-16T02:29:19.494006+0000","last_fullsized":"2024-09-16T02:29:19.494006+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:05.906602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015503139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494428+0000","last_change":"2024-09-16T02:29:19.494428+0000","last_active":"2024-09-16T02:29:19.494428+0000","last_peered":"2024-09-16T02:29:19.494428+0000","last_clean":"2024-09-16T02:29:19.494428+0000","last_became_active":"2024-09-16T02:29:17.482149+0000","last_became_peered":"2024-09-16T02:29:17.482149+0000","last_unstale":"2024-09-16T02:29:19.494428+0000","last_undegraded":"2024-09-16T02:29:19.494428+0000","last_fullsized":"2024-09-16T02:29:19.494428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:18.897041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.776174+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:19.776174+0000","last_peered":"2024-09-16T02:29:19.776174+0000","last_clean":"2024-09-16T02:29:19.776174+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:19.776174+0000","last_undegraded":"2024-09-16T02:29:19.776174+0000","last_fullsized":"2024-09-16T02:29:19.776174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493583+0000","last_change":"2024-09-16T02:29:19.493583+0000","last_active":"2024-09-16T02:29:19.493583+0000","last_peered":"2024-09-16T02:29:19.493583+0000","last_clean":"2024-09-16T02:29:19.493583+0000","last_became_active":"2024-09-16T02:29:17.482365+0000","last_became_peered":"2024-09-16T02:29:17.482365+0000","last_unstale":"2024-09-16T02:29:19.493583+0000","last_undegraded":"2024-09-16T02:29:19.493583+0000","last_fullsized":"2024-09-16T02:29:19.493583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:02.633320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493864+0000","last_change":"2024-09-16T02:29:19.494610+0000","last_active":"2024-09-16T02:29:19.493864+0000","last_peered":"2024-09-16T02:29:19.493864+0000","last_clean":"2024-09-16T02:29:19.493864+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:19.493864+0000","last_undegraded":"2024-09-16T02:29:19.493864+0000","last_fullsized":"2024-09-16T02:29:19.493864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:13:26.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020900630000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.506253+0000","last_change":"2024-09-16T02:29:19.506253+0000","last_active":"2024-09-16T02:29:19.506253+0000","last_peered":"2024-09-16T02:29:19.506253+0000","last_clean":"2024-09-16T02:29:19.506253+0000","last_became_active":"2024-09-16T02:29:17.486477+0000","last_became_peered":"2024-09-16T02:29:17.486477+0000","last_unstale":"2024-09-16T02:29:19.506253+0000","last_undegraded":"2024-09-16T02:29:19.506253+0000","last_fullsized":"2024-09-16T02:29:19.506253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:48:11.615610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493711+0000","last_change":"2024-09-16T02:29:19.493711+0000","last_active":"2024-09-16T02:29:19.493711+0000","last_peered":"2024-09-16T02:29:19.493711+0000","last_clean":"2024-09-16T02:29:19.493711+0000","last_became_active":"2024-09-16T02:29:17.482292+0000","last_became_peered":"2024-09-16T02:29:17.482292+0000","last_unstale":"2024-09-16T02:29:19.493711+0000","last_undegraded":"2024-09-16T02:29:19.493711+0000","last_fullsized":"2024-09-16T02:29:19.493711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:59.836546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.492730+0000","last_change":"2024-09-16T02:29:19.493362+0000","last_active":"2024-09-16T02:29:19.492730+0000","last_peered":"2024-09-16T02:29:19.492730+0000","last_clean":"2024-09-16T02:29:19.492730+0000","last_became_active":"2024-09-16T02:29:17.482879+0000","last_became_peered":"2024-09-16T02:29:17.482879+0000","last_unstale":"2024-09-16T02:29:19.492730+0000","last_undegraded":"2024-09-16T02:29:19.492730+0000","last_fullsized":"2024-09-16T02:29:19.492730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:38.603073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012747839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:23.370 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T02:29:23.370 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T02:29:23.370 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T02:29:23.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T02:29:23.687 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:23.687 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T02:29:23.704 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T02:29:23.704 INFO:teuthology.run_tasks:Running task osd_recovery.test_incomplete_pgs... 2024-09-16T02:29:23.717 INFO:tasks.osd_recovery:num_osds is 4 2024-09-16T02:29:23.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:29:24.000 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:24.001 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":16,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:29:19.474161+0000","last_up_change":"2024-09-16T02:29:11.729743+0000","last_in_change":"2024-09-16T02:29:01.784965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T02:29:16.348818+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":683366439}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":683366439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":683366439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":683366439}]},"public_addr":"172.21.15.137:6804/683366439","cluster_addr":"172.21.15.137:6805/683366439","heartbeat_back_addr":"172.21.15.137:6807/683366439","heartbeat_front_addr":"172.21.15.137:6806/683366439","state":["exists","up"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":4172164026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":4172164026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":4172164026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":4172164026}]},"public_addr":"172.21.15.137:6812/4172164026","cluster_addr":"172.21.15.137:6813/4172164026","heartbeat_back_addr":"172.21.15.137:6815/4172164026","heartbeat_front_addr":"172.21.15.137:6814/4172164026","state":["exists","up"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":2156154695}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":2156154695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":2156154695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":2156154695}]},"public_addr":"172.21.15.137:6808/2156154695","cluster_addr":"172.21.15.137:6809/2156154695","heartbeat_back_addr":"172.21.15.137:6811/2156154695","heartbeat_front_addr":"172.21.15.137:6810/2156154695","state":["exists","up"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6800","nonce":1497967517}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":1497967517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":1497967517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":1497967517}]},"public_addr":"172.21.15.137:6800/1497967517","cluster_addr":"172.21.15.137:6801/1497967517","heartbeat_back_addr":"172.21.15.137:6803/1497967517","heartbeat_front_addr":"172.21.15.137:6802/1497967517","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.492963+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.693906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.471476+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.155330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:29:24.019 INFO:tasks.osd_recovery.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-16T02:29:16.348818+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T02:29:24.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T02:29:24.317 INFO:teuthology.orchestra.run.smithi137.stdout:pg_num: 8 2024-09-16T02:29:24.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T02:29:24.620 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 16 2024-09-16T02:29:24.620 INFO:teuthology.orchestra.run.smithi137.stdout:fsid eabc9450-e776-4be8-8adb-00ce348fa2c2 2024-09-16T02:29:24.620 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-09-16T02:29:00.413885+0000 2024-09-16T02:29:24.620 INFO:teuthology.orchestra.run.smithi137.stdout:modified 2024-09-16T02:29:19.474161+0000 2024-09-16T02:29:24.621 INFO:teuthology.orchestra.run.smithi137.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T02:29:24.621 INFO:teuthology.orchestra.run.smithi137.stdout:crush_version 5 2024-09-16T02:29:24.621 INFO:teuthology.orchestra.run.smithi137.stdout:full_ratio 0.95 2024-09-16T02:29:24.621 INFO:teuthology.orchestra.run.smithi137.stdout:backfillfull_ratio 0.9 2024-09-16T02:29:24.621 INFO:teuthology.orchestra.run.smithi137.stdout:nearfull_ratio 0.85 2024-09-16T02:29:24.621 INFO:teuthology.orchestra.run.smithi137.stdout:require_min_compat_client luminous 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout:min_compat_client jewel 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout:require_osd_release squid 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout:stretch_mode_enabled false 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 16 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout: removed_snaps_queue [2~1] 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout:max_osd 4 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6804/683366439 v1:172.21.15.137:6805/683366439 exists,up 12c8d8a5-d99d-4bfc-99dd-7b19e351ed44 2024-09-16T02:29:24.622 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 up in weight 1 up_from 11 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6812/4172164026 v1:172.21.15.137:6813/4172164026 exists,up 8a607138-b9f7-49d3-941d-ca065ebe247f 2024-09-16T02:29:24.623 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 up in weight 1 up_from 11 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6808/2156154695 v1:172.21.15.137:6809/2156154695 exists,up 6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82 2024-09-16T02:29:24.623 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6800/1497967517 v1:172.21.15.137:6801/1497967517 exists,up 54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a 2024-09-16T02:29:24.636 INFO:tasks.osd_recovery.ceph_manager:['osd.0 up in weight 1 up_from 10 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6804/683366439 v1:172.21.15.137:6805/683366439 exists,up 12c8d8a5-d99d-4bfc-99dd-7b19e351ed44', 'osd.1 up in weight 1 up_from 11 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6812/4172164026 v1:172.21.15.137:6813/4172164026 exists,up 8a607138-b9f7-49d3-941d-ca065ebe247f', 'osd.2 up in weight 1 up_from 11 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6808/2156154695 v1:172.21.15.137:6809/2156154695 exists,up 6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82', 'osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6800/1497967517 v1:172.21.15.137:6801/1497967517 exists,up 54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a'] 2024-09-16T02:29:24.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T02:29:24.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T02:29:24.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T02:29:24.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T02:29:24.789 INFO:teuthology.orchestra.run.smithi137.stdout:38654705671 2024-09-16T02:29:24.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:29:24.791 INFO:teuthology.orchestra.run.smithi137.stdout:47244640263 2024-09-16T02:29:24.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:29:24.801 INFO:teuthology.orchestra.run.smithi137.stdout:47244640262 2024-09-16T02:29:24.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:29:24.803 INFO:teuthology.orchestra.run.smithi137.stdout:42949672966 2024-09-16T02:29:24.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:29:25.086 INFO:teuthology.orchestra.run.smithi137.stdout:47244640262 2024-09-16T02:29:25.086 INFO:teuthology.orchestra.run.smithi137.stdout:38654705670 2024-09-16T02:29:25.112 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640263 got 47244640262 for osd.2 2024-09-16T02:29:25.113 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705670 for osd.3 2024-09-16T02:29:25.138 INFO:teuthology.orchestra.run.smithi137.stdout:47244640261 2024-09-16T02:29:25.143 INFO:teuthology.orchestra.run.smithi137.stdout:42949672965 2024-09-16T02:29:25.158 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640262 got 47244640261 for osd.1 2024-09-16T02:29:25.162 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672966 got 42949672965 for osd.0 2024-09-16T02:29:26.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:29:26.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:29:26.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:29:26.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:29:26.420 INFO:teuthology.orchestra.run.smithi137.stdout:38654705671 2024-09-16T02:29:26.435 INFO:teuthology.orchestra.run.smithi137.stdout:47244640263 2024-09-16T02:29:26.438 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705671 for osd.3 2024-09-16T02:29:26.438 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:26.452 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640263 got 47244640263 for osd.2 2024-09-16T02:29:26.453 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:26.462 INFO:teuthology.orchestra.run.smithi137.stdout:47244640262 2024-09-16T02:29:26.463 INFO:teuthology.orchestra.run.smithi137.stdout:42949672966 2024-09-16T02:29:26.481 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640262 got 47244640262 for osd.1 2024-09-16T02:29:26.482 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:26.482 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672966 got 42949672966 for osd.0 2024-09-16T02:29:26.482 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:26.483 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2024-09-16T02:29:26.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:26.798 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:26.799 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:26.814 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T02:29:25.397502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108648,"kb_used_data":552,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378712,"statfs":{"total":386547056640,"available":386435801088,"internally_reserved":0,"allocated":565248,"data_stored":139806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000887"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494006+0000","last_change":"2024-09-16T02:29:19.494636+0000","last_active":"2024-09-16T02:29:19.494006+0000","last_peered":"2024-09-16T02:29:19.494006+0000","last_clean":"2024-09-16T02:29:19.494006+0000","last_became_active":"2024-09-16T02:29:17.477993+0000","last_became_peered":"2024-09-16T02:29:17.477993+0000","last_unstale":"2024-09-16T02:29:19.494006+0000","last_undegraded":"2024-09-16T02:29:19.494006+0000","last_fullsized":"2024-09-16T02:29:19.494006+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:05.906602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015503139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494428+0000","last_change":"2024-09-16T02:29:19.494428+0000","last_active":"2024-09-16T02:29:19.494428+0000","last_peered":"2024-09-16T02:29:19.494428+0000","last_clean":"2024-09-16T02:29:19.494428+0000","last_became_active":"2024-09-16T02:29:17.482149+0000","last_became_peered":"2024-09-16T02:29:17.482149+0000","last_unstale":"2024-09-16T02:29:19.494428+0000","last_undegraded":"2024-09-16T02:29:19.494428+0000","last_fullsized":"2024-09-16T02:29:19.494428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:18.897041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.776174+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:19.776174+0000","last_peered":"2024-09-16T02:29:19.776174+0000","last_clean":"2024-09-16T02:29:19.776174+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:19.776174+0000","last_undegraded":"2024-09-16T02:29:19.776174+0000","last_fullsized":"2024-09-16T02:29:19.776174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493583+0000","last_change":"2024-09-16T02:29:19.493583+0000","last_active":"2024-09-16T02:29:19.493583+0000","last_peered":"2024-09-16T02:29:19.493583+0000","last_clean":"2024-09-16T02:29:19.493583+0000","last_became_active":"2024-09-16T02:29:17.482365+0000","last_became_peered":"2024-09-16T02:29:17.482365+0000","last_unstale":"2024-09-16T02:29:19.493583+0000","last_undegraded":"2024-09-16T02:29:19.493583+0000","last_fullsized":"2024-09-16T02:29:19.493583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:02.633320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493864+0000","last_change":"2024-09-16T02:29:19.494610+0000","last_active":"2024-09-16T02:29:19.493864+0000","last_peered":"2024-09-16T02:29:19.493864+0000","last_clean":"2024-09-16T02:29:19.493864+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:19.493864+0000","last_undegraded":"2024-09-16T02:29:19.493864+0000","last_fullsized":"2024-09-16T02:29:19.493864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:13:26.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020900630000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.506253+0000","last_change":"2024-09-16T02:29:19.506253+0000","last_active":"2024-09-16T02:29:19.506253+0000","last_peered":"2024-09-16T02:29:19.506253+0000","last_clean":"2024-09-16T02:29:19.506253+0000","last_became_active":"2024-09-16T02:29:17.486477+0000","last_became_peered":"2024-09-16T02:29:17.486477+0000","last_unstale":"2024-09-16T02:29:19.506253+0000","last_undegraded":"2024-09-16T02:29:19.506253+0000","last_fullsized":"2024-09-16T02:29:19.506253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:48:11.615610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493711+0000","last_change":"2024-09-16T02:29:19.493711+0000","last_active":"2024-09-16T02:29:19.493711+0000","last_peered":"2024-09-16T02:29:19.493711+0000","last_clean":"2024-09-16T02:29:19.493711+0000","last_became_active":"2024-09-16T02:29:17.482292+0000","last_became_peered":"2024-09-16T02:29:17.482292+0000","last_unstale":"2024-09-16T02:29:19.493711+0000","last_undegraded":"2024-09-16T02:29:19.493711+0000","last_fullsized":"2024-09-16T02:29:19.493711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:59.836546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.492730+0000","last_change":"2024-09-16T02:29:19.493362+0000","last_active":"2024-09-16T02:29:19.492730+0000","last_peered":"2024-09-16T02:29:19.492730+0000","last_clean":"2024-09-16T02:29:19.492730+0000","last_became_active":"2024-09-16T02:29:17.482879+0000","last_became_peered":"2024-09-16T02:29:17.482879+0000","last_unstale":"2024-09-16T02:29:19.492730+0000","last_undegraded":"2024-09-16T02:29:19.492730+0000","last_fullsized":"2024-09-16T02:29:19.492730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:38.603073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012747839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:26.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:27.103 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:27.119 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T02:29:25.397502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108648,"kb_used_data":552,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378712,"statfs":{"total":386547056640,"available":386435801088,"internally_reserved":0,"allocated":565248,"data_stored":139806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":110618390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000887"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494006+0000","last_change":"2024-09-16T02:29:19.494636+0000","last_active":"2024-09-16T02:29:19.494006+0000","last_peered":"2024-09-16T02:29:19.494006+0000","last_clean":"2024-09-16T02:29:19.494006+0000","last_became_active":"2024-09-16T02:29:17.477993+0000","last_became_peered":"2024-09-16T02:29:17.477993+0000","last_unstale":"2024-09-16T02:29:19.494006+0000","last_undegraded":"2024-09-16T02:29:19.494006+0000","last_fullsized":"2024-09-16T02:29:19.494006+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:05.906602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015503139999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.494428+0000","last_change":"2024-09-16T02:29:19.494428+0000","last_active":"2024-09-16T02:29:19.494428+0000","last_peered":"2024-09-16T02:29:19.494428+0000","last_clean":"2024-09-16T02:29:19.494428+0000","last_became_active":"2024-09-16T02:29:17.482149+0000","last_became_peered":"2024-09-16T02:29:17.482149+0000","last_unstale":"2024-09-16T02:29:19.494428+0000","last_undegraded":"2024-09-16T02:29:19.494428+0000","last_fullsized":"2024-09-16T02:29:19.494428+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:18.897041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.776174+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:19.776174+0000","last_peered":"2024-09-16T02:29:19.776174+0000","last_clean":"2024-09-16T02:29:19.776174+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:19.776174+0000","last_undegraded":"2024-09-16T02:29:19.776174+0000","last_fullsized":"2024-09-16T02:29:19.776174+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493583+0000","last_change":"2024-09-16T02:29:19.493583+0000","last_active":"2024-09-16T02:29:19.493583+0000","last_peered":"2024-09-16T02:29:19.493583+0000","last_clean":"2024-09-16T02:29:19.493583+0000","last_became_active":"2024-09-16T02:29:17.482365+0000","last_became_peered":"2024-09-16T02:29:17.482365+0000","last_unstale":"2024-09-16T02:29:19.493583+0000","last_undegraded":"2024-09-16T02:29:19.493583+0000","last_fullsized":"2024-09-16T02:29:19.493583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:02.633320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493864+0000","last_change":"2024-09-16T02:29:19.494610+0000","last_active":"2024-09-16T02:29:19.493864+0000","last_peered":"2024-09-16T02:29:19.493864+0000","last_clean":"2024-09-16T02:29:19.493864+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:19.493864+0000","last_undegraded":"2024-09-16T02:29:19.493864+0000","last_fullsized":"2024-09-16T02:29:19.493864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:13:26.120624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020900630000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.506253+0000","last_change":"2024-09-16T02:29:19.506253+0000","last_active":"2024-09-16T02:29:19.506253+0000","last_peered":"2024-09-16T02:29:19.506253+0000","last_clean":"2024-09-16T02:29:19.506253+0000","last_became_active":"2024-09-16T02:29:17.486477+0000","last_became_peered":"2024-09-16T02:29:17.486477+0000","last_unstale":"2024-09-16T02:29:19.506253+0000","last_undegraded":"2024-09-16T02:29:19.506253+0000","last_fullsized":"2024-09-16T02:29:19.506253+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:48:11.615610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.493711+0000","last_change":"2024-09-16T02:29:19.493711+0000","last_active":"2024-09-16T02:29:19.493711+0000","last_peered":"2024-09-16T02:29:19.493711+0000","last_clean":"2024-09-16T02:29:19.493711+0000","last_became_active":"2024-09-16T02:29:17.482292+0000","last_became_peered":"2024-09-16T02:29:17.482292+0000","last_unstale":"2024-09-16T02:29:19.493711+0000","last_undegraded":"2024-09-16T02:29:19.493711+0000","last_fullsized":"2024-09-16T02:29:19.493711+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:08:59.836546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T02:29:19.492730+0000","last_change":"2024-09-16T02:29:19.493362+0000","last_active":"2024-09-16T02:29:19.492730+0000","last_peered":"2024-09-16T02:29:19.492730+0000","last_clean":"2024-09-16T02:29:19.492730+0000","last_became_active":"2024-09-16T02:29:17.482879+0000","last_became_peered":"2024-09-16T02:29:17.482879+0000","last_unstale":"2024-09-16T02:29:19.492730+0000","last_undegraded":"2024-09-16T02:29:19.492730+0000","last_fullsized":"2024-09-16T02:29:19.492730+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:38.603073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012747839999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672966,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":34961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":34942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:27.120 INFO:tasks.osd_recovery.ceph_manager:clean! 2024-09-16T02:29:27.120 INFO:tasks.osd_recovery:Testing incomplete pgs... 2024-09-16T02:29:27.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set osd_recovery_delay_start 1000 2024-09-16T02:29:27.233 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:29:27.233 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-16T02:29:27.234 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:29:27.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_recovery_delay_start 1000 2024-09-16T02:29:27.366 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:29:27.366 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-16T02:29:27.366 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:29:27.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set osd_recovery_delay_start 1000 2024-09-16T02:29:27.497 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:29:27.497 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-16T02:29:27.497 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:29:27.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set osd_recovery_delay_start 1000 2024-09-16T02:29:27.616 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:29:27.616 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-16T02:29:27.616 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:29:27.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 1 2024-09-16T02:29:27.925 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-09-16T02:29:27.919+0000 7f36593c7640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-16T02:29:28.519 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.0. marked out osd.1. 2024-09-16T02:29:28.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T02:29:28.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T02:29:28.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T02:29:28.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T02:29:28.707 INFO:teuthology.orchestra.run.smithi137.stdout:47244640264 2024-09-16T02:29:28.708 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:28.709 INFO:teuthology.orchestra.run.smithi137.stdout:42949672968 2024-09-16T02:29:28.710 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:28.710 INFO:teuthology.orchestra.run.smithi137.stdout:38654705673 2024-09-16T02:29:28.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:29:28.712 INFO:teuthology.orchestra.run.smithi137.stdout:47244640265 2024-09-16T02:29:28.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:29:29.037 INFO:teuthology.orchestra.run.smithi137.stdout:47244640263 2024-09-16T02:29:29.046 INFO:teuthology.orchestra.run.smithi137.stdout:38654705672 2024-09-16T02:29:29.056 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640265 got 47244640263 for osd.2 2024-09-16T02:29:29.065 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705673 got 38654705672 for osd.3 2024-09-16T02:29:30.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:29:30.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:29:30.370 INFO:teuthology.orchestra.run.smithi137.stdout:47244640265 2024-09-16T02:29:30.389 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640265 got 47244640265 for osd.2 2024-09-16T02:29:30.389 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:30.404 INFO:teuthology.orchestra.run.smithi137.stdout:38654705673 2024-09-16T02:29:30.422 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705673 got 38654705673 for osd.3 2024-09-16T02:29:30.422 DEBUG:teuthology.parallel:result is None 2024-09-16T02:29:30.422 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2024-09-16T02:29:30.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:30.753 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:30.754 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:30.770 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-16T02:29:29.530737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54336,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689344,"statfs":{"total":193273528320,"available":193217888256,"internally_reserved":0,"allocated":294912,"data_stored":75248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.133777"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:28.519030+0000","last_change":"2024-09-16T02:29:28.519030+0000","last_active":"2024-09-16T02:29:16.448904+0000","last_peered":"2024-09-16T02:29:16.448904+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T02:29:28.519030+0000","last_undegraded":"2024-09-16T02:29:28.519030+0000","last_fullsized":"2024-09-16T02:29:28.519030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.233983+0000","last_change":"2024-09-16T02:29:28.522100+0000","last_active":"2024-09-16T02:29:28.522100+0000","last_peered":"2024-09-16T02:29:25.233983+0000","last_clean":"2024-09-16T02:29:25.233983+0000","last_became_active":"2024-09-16T02:29:17.482149+0000","last_became_peered":"2024-09-16T02:29:17.482149+0000","last_unstale":"2024-09-16T02:29:25.233983+0000","last_undegraded":"2024-09-16T02:29:25.233983+0000","last_fullsized":"2024-09-16T02:29:25.233983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T02:29:28.523408+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:28.523408+0000","last_peered":"2024-09-16T02:29:28.523408+0000","last_clean":"2024-09-16T02:29:28.523408+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:28.523408+0000","last_undegraded":"2024-09-16T02:29:28.523408+0000","last_fullsized":"2024-09-16T02:29:28.523408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.234405+0000","last_change":"2024-09-16T02:29:28.519401+0000","last_active":"2024-09-16T02:29:28.519401+0000","last_peered":"2024-09-16T02:29:25.234405+0000","last_clean":"2024-09-16T02:29:25.234405+0000","last_became_active":"2024-09-16T02:29:17.482365+0000","last_became_peered":"2024-09-16T02:29:17.482365+0000","last_unstale":"2024-09-16T02:29:25.234405+0000","last_undegraded":"2024-09-16T02:29:25.234405+0000","last_fullsized":"2024-09-16T02:29:25.234405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":8,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:28.521461+0000","last_change":"2024-09-16T02:29:28.521461+0000","last_active":"2024-09-16T02:29:28.521461+0000","last_peered":"2024-09-16T02:29:17.485100+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:28.521461+0000","last_undegraded":"2024-09-16T02:29:28.521461+0000","last_fullsized":"2024-09-16T02:29:28.521461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":2,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:25.235394+0000","last_change":"2024-09-16T02:29:28.686755+0000","last_active":"2024-09-16T02:29:28.686755+0000","last_peered":"2024-09-16T02:29:25.235394+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:17.486477+0000","last_became_peered":"2024-09-16T02:29:17.486477+0000","last_unstale":"2024-09-16T02:29:25.235394+0000","last_undegraded":"2024-09-16T02:29:25.235394+0000","last_fullsized":"2024-09-16T02:29:25.235394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.233969+0000","last_change":"2024-09-16T02:29:28.519194+0000","last_active":"2024-09-16T02:29:28.519194+0000","last_peered":"2024-09-16T02:29:25.233969+0000","last_clean":"2024-09-16T02:29:25.233969+0000","last_became_active":"2024-09-16T02:29:17.482292+0000","last_became_peered":"2024-09-16T02:29:17.482292+0000","last_unstale":"2024-09-16T02:29:25.233969+0000","last_undegraded":"2024-09-16T02:29:25.233969+0000","last_fullsized":"2024-09-16T02:29:25.233969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":2,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:17.471329+0000","last_change":"2024-09-16T02:29:28.687329+0000","last_active":"2024-09-16T02:29:16.448904+0000","last_peered":"2024-09-16T02:29:16.448904+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T02:29:17.471329+0000","last_undegraded":"2024-09-16T02:29:17.471329+0000","last_fullsized":"2024-09-16T02:29:17.471329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":37624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640264,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672968,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":37624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:30.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:31.065 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:31.066 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:31.084 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-16T02:29:29.530737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54336,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689344,"statfs":{"total":193273528320,"available":193217888256,"internally_reserved":0,"allocated":294912,"data_stored":75248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.133777"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:28.519030+0000","last_change":"2024-09-16T02:29:28.519030+0000","last_active":"2024-09-16T02:29:16.448904+0000","last_peered":"2024-09-16T02:29:16.448904+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T02:29:28.519030+0000","last_undegraded":"2024-09-16T02:29:28.519030+0000","last_fullsized":"2024-09-16T02:29:28.519030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.233983+0000","last_change":"2024-09-16T02:29:28.522100+0000","last_active":"2024-09-16T02:29:28.522100+0000","last_peered":"2024-09-16T02:29:25.233983+0000","last_clean":"2024-09-16T02:29:25.233983+0000","last_became_active":"2024-09-16T02:29:17.482149+0000","last_became_peered":"2024-09-16T02:29:17.482149+0000","last_unstale":"2024-09-16T02:29:25.233983+0000","last_undegraded":"2024-09-16T02:29:25.233983+0000","last_fullsized":"2024-09-16T02:29:25.233983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T02:29:28.523408+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:28.523408+0000","last_peered":"2024-09-16T02:29:28.523408+0000","last_clean":"2024-09-16T02:29:28.523408+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:28.523408+0000","last_undegraded":"2024-09-16T02:29:28.523408+0000","last_fullsized":"2024-09-16T02:29:28.523408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.234405+0000","last_change":"2024-09-16T02:29:28.519401+0000","last_active":"2024-09-16T02:29:28.519401+0000","last_peered":"2024-09-16T02:29:25.234405+0000","last_clean":"2024-09-16T02:29:25.234405+0000","last_became_active":"2024-09-16T02:29:17.482365+0000","last_became_peered":"2024-09-16T02:29:17.482365+0000","last_unstale":"2024-09-16T02:29:25.234405+0000","last_undegraded":"2024-09-16T02:29:25.234405+0000","last_fullsized":"2024-09-16T02:29:25.234405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":8,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:28.521461+0000","last_change":"2024-09-16T02:29:28.521461+0000","last_active":"2024-09-16T02:29:28.521461+0000","last_peered":"2024-09-16T02:29:17.485100+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:28.521461+0000","last_undegraded":"2024-09-16T02:29:28.521461+0000","last_fullsized":"2024-09-16T02:29:28.521461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":2,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:25.235394+0000","last_change":"2024-09-16T02:29:28.686755+0000","last_active":"2024-09-16T02:29:28.686755+0000","last_peered":"2024-09-16T02:29:25.235394+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:17.486477+0000","last_became_peered":"2024-09-16T02:29:17.486477+0000","last_unstale":"2024-09-16T02:29:25.235394+0000","last_undegraded":"2024-09-16T02:29:25.235394+0000","last_fullsized":"2024-09-16T02:29:25.235394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.233969+0000","last_change":"2024-09-16T02:29:28.519194+0000","last_active":"2024-09-16T02:29:28.519194+0000","last_peered":"2024-09-16T02:29:25.233969+0000","last_clean":"2024-09-16T02:29:25.233969+0000","last_became_active":"2024-09-16T02:29:17.482292+0000","last_became_peered":"2024-09-16T02:29:17.482292+0000","last_unstale":"2024-09-16T02:29:25.233969+0000","last_undegraded":"2024-09-16T02:29:25.233969+0000","last_fullsized":"2024-09-16T02:29:25.233969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":2,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:17.471329+0000","last_change":"2024-09-16T02:29:28.687329+0000","last_active":"2024-09-16T02:29:16.448904+0000","last_peered":"2024-09-16T02:29:16.448904+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T02:29:17.471329+0000","last_undegraded":"2024-09-16T02:29:17.471329+0000","last_fullsized":"2024-09-16T02:29:17.471329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":37624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640264,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672968,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":37624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:31.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:31.367 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:31.367 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:31.383 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-16T02:29:29.530737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54336,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689344,"statfs":{"total":193273528320,"available":193217888256,"internally_reserved":0,"allocated":294912,"data_stored":75248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.133777"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":4,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:28.519030+0000","last_change":"2024-09-16T02:29:28.519030+0000","last_active":"2024-09-16T02:29:16.448904+0000","last_peered":"2024-09-16T02:29:16.448904+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T02:29:28.519030+0000","last_undegraded":"2024-09-16T02:29:28.519030+0000","last_fullsized":"2024-09-16T02:29:28.519030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.233983+0000","last_change":"2024-09-16T02:29:28.522100+0000","last_active":"2024-09-16T02:29:28.522100+0000","last_peered":"2024-09-16T02:29:25.233983+0000","last_clean":"2024-09-16T02:29:25.233983+0000","last_became_active":"2024-09-16T02:29:17.482149+0000","last_became_peered":"2024-09-16T02:29:17.482149+0000","last_unstale":"2024-09-16T02:29:25.233983+0000","last_undegraded":"2024-09-16T02:29:25.233983+0000","last_fullsized":"2024-09-16T02:29:25.233983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T02:29:28.523408+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:28.523408+0000","last_peered":"2024-09-16T02:29:28.523408+0000","last_clean":"2024-09-16T02:29:28.523408+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:28.523408+0000","last_undegraded":"2024-09-16T02:29:28.523408+0000","last_fullsized":"2024-09-16T02:29:28.523408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.234405+0000","last_change":"2024-09-16T02:29:28.519401+0000","last_active":"2024-09-16T02:29:28.519401+0000","last_peered":"2024-09-16T02:29:25.234405+0000","last_clean":"2024-09-16T02:29:25.234405+0000","last_became_active":"2024-09-16T02:29:17.482365+0000","last_became_peered":"2024-09-16T02:29:17.482365+0000","last_unstale":"2024-09-16T02:29:25.234405+0000","last_undegraded":"2024-09-16T02:29:25.234405+0000","last_fullsized":"2024-09-16T02:29:25.234405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":8,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:28.521461+0000","last_change":"2024-09-16T02:29:28.521461+0000","last_active":"2024-09-16T02:29:28.521461+0000","last_peered":"2024-09-16T02:29:17.485100+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:28.521461+0000","last_undegraded":"2024-09-16T02:29:28.521461+0000","last_fullsized":"2024-09-16T02:29:28.521461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":2,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:25.235394+0000","last_change":"2024-09-16T02:29:28.686755+0000","last_active":"2024-09-16T02:29:28.686755+0000","last_peered":"2024-09-16T02:29:25.235394+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:17.486477+0000","last_became_peered":"2024-09-16T02:29:17.486477+0000","last_unstale":"2024-09-16T02:29:25.235394+0000","last_undegraded":"2024-09-16T02:29:25.235394+0000","last_fullsized":"2024-09-16T02:29:25.235394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":23,"reported_epoch":17,"state":"peering","last_fresh":"2024-09-16T02:29:25.233969+0000","last_change":"2024-09-16T02:29:28.519194+0000","last_active":"2024-09-16T02:29:28.519194+0000","last_peered":"2024-09-16T02:29:25.233969+0000","last_clean":"2024-09-16T02:29:25.233969+0000","last_became_active":"2024-09-16T02:29:17.482292+0000","last_became_peered":"2024-09-16T02:29:17.482292+0000","last_unstale":"2024-09-16T02:29:25.233969+0000","last_undegraded":"2024-09-16T02:29:25.233969+0000","last_fullsized":"2024-09-16T02:29:25.233969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":2,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:17.471329+0000","last_change":"2024-09-16T02:29:28.687329+0000","last_active":"2024-09-16T02:29:16.448904+0000","last_peered":"2024-09-16T02:29:16.448904+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T02:29:17.471329+0000","last_undegraded":"2024-09-16T02:29:17.471329+0000","last_fullsized":"2024-09-16T02:29:17.471329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":37624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640264,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672968,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":37624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 18, 'state': 'peering', 'last_fresh': '2024-09-16T02:29:28.519030+0000', 'last_change': '2024-09-16T02:29:28.519030+0000', 'last_active': '2024-09-16T02:29:16.448904+0000', 'last_peered': '2024-09-16T02:29:16.448904+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T02:29:28.519030+0000', 'last_undegraded': '2024-09-16T02:29:28.519030+0000', 'last_fullsized': '2024-09-16T02:29:28.519030+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 17, 'state': 'peering', 'last_fresh': '2024-09-16T02:29:25.233983+0000', 'last_change': '2024-09-16T02:29:28.522100+0000', 'last_active': '2024-09-16T02:29:28.522100+0000', 'last_peered': '2024-09-16T02:29:25.233983+0000', 'last_clean': '2024-09-16T02:29:25.233983+0000', 'last_became_active': '2024-09-16T02:29:17.482149+0000', 'last_became_peered': '2024-09-16T02:29:17.482149+0000', 'last_unstale': '2024-09-16T02:29:25.233983+0000', 'last_undegraded': '2024-09-16T02:29:25.233983+0000', 'last_fullsized': '2024-09-16T02:29:25.233983+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 17, 'state': 'peering', 'last_fresh': '2024-09-16T02:29:25.234405+0000', 'last_change': '2024-09-16T02:29:28.519401+0000', 'last_active': '2024-09-16T02:29:28.519401+0000', 'last_peered': '2024-09-16T02:29:25.234405+0000', 'last_clean': '2024-09-16T02:29:25.234405+0000', 'last_became_active': '2024-09-16T02:29:17.482365+0000', 'last_became_peered': '2024-09-16T02:29:17.482365+0000', 'last_unstale': '2024-09-16T02:29:25.234405+0000', 'last_undegraded': '2024-09-16T02:29:25.234405+0000', 'last_fullsized': '2024-09-16T02:29:25.234405+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:29:31.384 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 8, 'reported_epoch': 18, 'state': 'peering', 'last_fresh': '2024-09-16T02:29:28.521461+0000', 'last_change': '2024-09-16T02:29:28.521461+0000', 'last_active': '2024-09-16T02:29:28.521461+0000', 'last_peered': '2024-09-16T02:29:17.485100+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:17.484954+0000', 'last_became_peered': '2024-09-16T02:29:17.484954+0000', 'last_unstale': '2024-09-16T02:29:28.521461+0000', 'last_undegraded': '2024-09-16T02:29:28.521461+0000', 'last_fullsized': '2024-09-16T02:29:28.521461+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:31.385 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:29:31.385 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 2, 'reported_epoch': 18, 'state': 'peering', 'last_fresh': '2024-09-16T02:29:25.235394+0000', 'last_change': '2024-09-16T02:29:28.686755+0000', 'last_active': '2024-09-16T02:29:28.686755+0000', 'last_peered': '2024-09-16T02:29:25.235394+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:17.486477+0000', 'last_became_peered': '2024-09-16T02:29:17.486477+0000', 'last_unstale': '2024-09-16T02:29:25.235394+0000', 'last_undegraded': '2024-09-16T02:29:25.235394+0000', 'last_fullsized': '2024-09-16T02:29:25.235394+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:31.385 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-16T02:29:31.385 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 17, 'state': 'peering', 'last_fresh': '2024-09-16T02:29:25.233969+0000', 'last_change': '2024-09-16T02:29:28.519194+0000', 'last_active': '2024-09-16T02:29:28.519194+0000', 'last_peered': '2024-09-16T02:29:25.233969+0000', 'last_clean': '2024-09-16T02:29:25.233969+0000', 'last_became_active': '2024-09-16T02:29:17.482292+0000', 'last_became_peered': '2024-09-16T02:29:17.482292+0000', 'last_unstale': '2024-09-16T02:29:25.233969+0000', 'last_undegraded': '2024-09-16T02:29:25.233969+0000', 'last_fullsized': '2024-09-16T02:29:25.233969+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001711783, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:31.385 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:29:31.385 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 2, 'reported_epoch': 18, 'state': 'peering', 'last_fresh': '2024-09-16T02:29:17.471329+0000', 'last_change': '2024-09-16T02:29:28.687329+0000', 'last_active': '2024-09-16T02:29:16.448904+0000', 'last_peered': '2024-09-16T02:29:16.448904+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T02:29:17.471329+0000', 'last_undegraded': '2024-09-16T02:29:17.471329+0000', 'last_fullsized': '2024-09-16T02:29:17.471329+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:31.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:29:31.727 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:31.728 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":31,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55656448,"bytes_avail":193217871872,"bytes_total":193273528320,"inactive_pgs_ratio":0.25,"misplaced_objects":2,"misplaced_total":4,"misplaced_ratio":0.5},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:29:31.744 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:29:31.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:32.033 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:32.033 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:32.051 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-16T02:29:31.399075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54352,"kb_used_data":304,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689328,"statfs":{"total":193273528320,"available":193217871872,"internally_reserved":0,"allocated":311296,"data_stored":80779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.238499"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T02:29:28.523408+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:28.523408+0000","last_peered":"2024-09-16T02:29:28.523408+0000","last_clean":"2024-09-16T02:29:28.523408+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:28.523408+0000","last_undegraded":"2024-09-16T02:29:28.523408+0000","last_fullsized":"2024-09-16T02:29:28.523408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":8,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:28.521461+0000","last_change":"2024-09-16T02:29:28.521461+0000","last_active":"2024-09-16T02:29:28.521461+0000","last_peered":"2024-09-16T02:29:17.485100+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:17.484954+0000","last_became_peered":"2024-09-16T02:29:17.484954+0000","last_unstale":"2024-09-16T02:29:28.521461+0000","last_undegraded":"2024-09-16T02:29:28.521461+0000","last_fullsized":"2024-09-16T02:29:28.521461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":2,"reported_epoch":18,"state":"peering","last_fresh":"2024-09-16T02:29:17.471329+0000","last_change":"2024-09-16T02:29:28.687329+0000","last_active":"2024-09-16T02:29:16.448904+0000","last_peered":"2024-09-16T02:29:16.448904+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T02:29:17.471329+0000","last_undegraded":"2024-09-16T02:29:17.471329+0000","last_fullsized":"2024-09-16T02:29:17.471329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":37624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640264,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672968,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:35.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:35.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:35.377 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:35.393 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-16T02:29:33.399482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001980"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640265,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672969,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:35.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:35.693 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-09-16T02:29:35.400096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002155"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640265,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672969,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:35.711 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:29:35.711 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:35.711 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:29:35.711 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:35.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:29:36.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:36.049 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":35,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:29:36.065 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:29:36.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:36.342 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:36.342 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:36.358 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-09-16T02:29:35.400096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002155"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640265,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672969,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:39.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:39.670 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:39.670 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:39.688 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-09-16T02:29:39.401004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002528"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640266,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672970,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:39.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:39.978 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:39.978 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:39.994 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-09-16T02:29:39.401004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002528"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640266,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672970,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:39.995 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:29:39.995 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:39.995 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:29:39.995 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:39.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:29:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:40.324 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":39,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:29:40.341 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:29:40.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:40.620 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:40.620 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:40.637 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-09-16T02:29:39.401004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002528"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640266,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672970,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:43.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:43.949 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:43.949 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:43.967 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-16T02:29:43.402111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003035"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672971,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:43.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:44.252 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:44.269 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-16T02:29:43.402111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003035"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672971,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:44.270 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:29:44.270 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:44.270 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:29:44.270 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:44.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:29:44.619 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:44.619 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":44,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:29:44.637 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:29:44.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:44.930 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:44.931 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:44.948 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-16T02:29:43.402111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003035"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672971,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:47.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:48.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:48.254 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:48.271 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-16T02:29:47.403208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:48.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:48.560 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:48.560 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:48.578 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-16T02:29:47.403208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:48.578 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:29:48.579 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:48.579 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:29:48.579 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:48.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:29:48.938 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:48.938 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":48,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:29:48.957 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:29:48.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:49.267 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:49.267 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:49.284 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-16T02:29:47.403208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:52.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:52.596 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:52.596 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:52.614 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-16T02:29:51.404310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003306"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640269,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640268,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:52.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:52.915 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:52.915 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:52.933 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-16T02:29:51.404310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003306"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640269,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640268,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:52.934 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:29:52.934 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:52.934 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:29:52.934 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:52.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:29:53.283 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:53.283 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":52,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:29:53.301 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:29:53.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:53.573 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:53.573 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:53.590 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-09-16T02:29:53.404849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640269,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672973,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:56.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:56.899 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:56.914 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-16T02:29:55.405382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640269,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672973,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:56.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:57.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:57.210 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:57.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-16T02:29:55.405382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640269,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672973,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:29:57.228 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:29:57.228 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:29:57.228 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:29:57.229 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:29:57.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:29:57.598 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:57.598 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":57,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:29:57.616 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:29:57.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:29:57.884 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:29:57.885 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:29:57.901 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-09-16T02:29:57.405878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003207"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640269,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672974,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:00.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:01.198 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:01.199 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:01.214 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-16T02:29:59.406484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003275"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640271,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640270,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672974,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:01.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:01.513 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:01.514 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:01.529 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-16T02:30:01.406983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003219"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640271,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640270,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672974,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:01.530 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:01.530 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:01.531 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:01.531 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:01.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:01.871 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:01.871 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":61,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:01.888 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:01.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:02.158 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:02.158 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:02.175 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-16T02:30:01.406983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003219"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640271,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640270,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672974,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:05.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:05.477 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:05.478 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:05.495 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-16T02:30:05.407849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640271,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672975,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:05.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:05.778 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:05.794 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-16T02:30:05.407849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640271,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672975,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:05.795 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:05.795 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:05.795 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:05.795 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:05.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:06.155 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:06.155 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":65,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T02:29:00.415687+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:06.174 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:06.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:06.470 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:06.470 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:06.486 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-16T02:30:05.407849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640271,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672975,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:09.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:09.791 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:09.791 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:09.809 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-16T02:30:09.408880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640273,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640272,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672976,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:09.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:10.103 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:10.103 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:10.122 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-16T02:30:09.408880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640273,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640272,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672976,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:10.122 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:10.123 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:10.123 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:10.123 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:10.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:10.467 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:10.467 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":70,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-16T02:30:07.408381+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:30:10.485 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:10.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:10.754 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:10.755 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:10.770 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-16T02:30:09.408880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640273,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640272,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672976,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:13.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:14.075 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:14.075 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:14.091 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-09-16T02:30:13.410013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672977,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:14.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:14.383 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:14.383 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:14.399 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-09-16T02:30:13.410013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672977,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:14.399 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:14.399 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:14.400 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:14.400 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:14.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:14.763 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:14.763 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":74,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:14.781 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:14.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:15.051 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:15.051 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:15.067 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-09-16T02:30:13.410013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672977,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:18.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:18.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:18.377 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:18.393 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-09-16T02:30:17.410997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:18.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:18.684 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:18.684 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:18.700 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-09-16T02:30:17.410997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:18.701 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:18.701 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:18.701 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:18.701 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:18.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:19.046 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":78,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:19.063 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:19.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:19.353 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:19.353 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:19.370 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-09-16T02:30:17.410997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640273,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:22.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:22.682 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:22.698 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-16T02:30:21.411875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640275,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640274,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:22.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:22.996 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:22.996 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:23.013 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-16T02:30:21.411875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640275,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640274,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:23.013 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:23.013 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:23.014 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:23.014 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:23.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:23.356 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:23.356 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":82,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:23.373 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:23.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:23.670 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:23.671 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:23.688 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-16T02:30:23.412333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002898"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640276,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640275,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:26.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:27.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:27.004 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:27.022 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-16T02:30:25.412658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002644"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640276,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640275,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:27.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:27.304 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:27.304 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:27.320 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-16T02:30:25.412658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002644"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640276,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640275,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:27.321 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:27.321 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:27.321 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:27.321 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:27.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:27.656 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:27.656 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":87,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:27.673 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:27.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:27.963 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:27.964 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:27.981 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2024-09-16T02:30:27.413123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002585"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640276,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640275,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672980,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:30.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:31.288 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:31.288 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:31.303 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-09-16T02:30:29.413639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002641"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640277,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640276,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672980,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:31.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:31.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:31.606 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:31.624 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-16T02:30:31.414161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640277,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640276,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672980,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:31.625 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:31.625 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:31.625 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:31.625 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:31.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:31.980 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:31.980 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":91,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:31.997 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:31.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:32.275 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:32.292 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-16T02:30:31.414161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640277,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640276,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672980,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:35.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:35.605 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:35.606 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:35.624 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-16T02:30:35.415182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002848"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640278,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672981,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:35.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:35.909 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:35.909 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:35.925 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-16T02:30:35.415182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002848"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640278,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672981,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:35.926 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:35.926 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:35.927 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:35.927 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:35.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:36.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:36.262 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":95,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:36.279 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:36.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:36.576 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:36.576 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:36.593 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-16T02:30:35.415182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002848"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640278,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672981,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:39.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:39.905 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:39.922 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-16T02:30:39.416142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640278,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672982,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:39.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:40.218 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:40.218 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:40.234 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-16T02:30:39.416142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640278,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672982,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:40.235 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:40.235 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:40.236 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:40.236 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:40.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:40.604 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:40.604 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":100,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:40.625 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:40.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:40.905 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:40.906 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:40.921 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-16T02:30:39.416142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640278,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672982,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:43.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:44.232 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:44.249 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-16T02:30:43.417454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003293"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672983,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:44.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:44.545 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:44.546 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:44.563 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-16T02:30:43.417454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003293"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672983,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:44.564 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:44.564 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:44.565 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:44.565 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:44.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:44.911 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:44.911 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":104,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:44.929 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:44.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:45.209 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:45.209 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:45.225 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-16T02:30:43.417454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003293"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672983,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:48.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:48.535 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:48.536 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:48.553 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-16T02:30:47.418702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003520"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:48.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:48.850 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:48.850 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:48.867 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-16T02:30:47.418702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003520"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:48.868 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:48.868 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:48.868 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:48.868 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:48.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:49.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:49.212 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":108,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:49.228 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:49.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:49.515 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:49.515 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:49.532 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-16T02:30:49.419372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003788"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640280,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:52.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:52.847 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:52.847 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:52.865 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-16T02:30:51.419953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003810"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640280,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:52.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:53.154 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:53.155 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:53.170 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-16T02:30:51.419953+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003810"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640281,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640280,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:53.171 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:53.171 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:53.172 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:53.172 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:53.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:53.515 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:53.515 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":113,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:53.533 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:53.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:53.821 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:53.822 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:53.839 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-16T02:30:53.420466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003695"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640282,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640281,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672985,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:56.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:57.151 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:57.151 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:57.169 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-16T02:30:55.421052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003597"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640282,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640281,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672985,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:57.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:57.469 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:57.469 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:57.488 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-09-16T02:30:57.421643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003523"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640282,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640281,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:30:57.489 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:30:57.489 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:30:57.489 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:30:57.489 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:30:57.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:30:57.843 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:57.843 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":117,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:30:57.861 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:30:57.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:30:58.159 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:30:58.159 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:30:58.177 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-09-16T02:30:57.421643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003523"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640282,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640281,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:01.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:01.482 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:01.483 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:01.499 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-09-16T02:31:01.422808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003435"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640282,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:01.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:01.787 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:01.787 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:01.804 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-09-16T02:31:01.422808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003435"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640282,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:01.804 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:01.805 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:01.805 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:01.805 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:01.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:02.143 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:02.143 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":121,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:02.160 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:02.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:02.450 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:02.468 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-09-16T02:31:01.422808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003435"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640282,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:05.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:05.779 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:05.796 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-16T02:31:05.423809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003343"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640283,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:05.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:06.096 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:06.097 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:06.115 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-16T02:31:05.423809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003343"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640283,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:06.115 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:06.115 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:06.116 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:06.116 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:06.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:06.474 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:06.475 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":126,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:06.493 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:06.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:06.781 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:06.781 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:06.797 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-16T02:31:05.423809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003343"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640284,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640283,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672987,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:09.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:10.101 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:10.101 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:10.117 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-16T02:31:09.424833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003189"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640285,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640284,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672988,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:10.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:10.397 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:10.397 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:10.414 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-16T02:31:09.424833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003189"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640285,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640284,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672988,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:10.415 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:10.415 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:10.415 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:10.415 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:10.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:10.783 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:10.783 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":130,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:10.801 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:10.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:11.069 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:11.069 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:11.086 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-16T02:31:09.424833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003189"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640285,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640284,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672988,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:14.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:14.398 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:14.398 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:14.416 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-16T02:31:13.426029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003220"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640286,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640285,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:14.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:14.715 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:14.715 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:14.731 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-16T02:31:13.426029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003220"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640286,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640285,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:14.732 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:14.733 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:14.733 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:14.733 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:14.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:15.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:15.073 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":134,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:15.089 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:15.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:15.375 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:15.375 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:15.391 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-16T02:31:13.426029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003220"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640286,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640285,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:18.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:18.707 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:18.707 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:18.725 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-16T02:31:17.426870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003060"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640286,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640285,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672990,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:18.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:19.002 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:19.019 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-16T02:31:17.426870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003060"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640286,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640285,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672990,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:19.019 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:19.019 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:19.020 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:19.020 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:19.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:19.385 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:19.385 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":138,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:19.404 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:19.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:19.691 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:19.707 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-16T02:31:19.427358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003081"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640287,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640286,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672990,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:22.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:23.014 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:23.015 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:23.031 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-16T02:31:21.427740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002907"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640287,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640286,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672990,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:23.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:23.313 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:23.329 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-16T02:31:21.427740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002907"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":26,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.535134+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.535134+0000","last_peered":"2024-09-16T02:29:29.535134+0000","last_clean":"2024-09-16T02:29:29.535134+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.535134+0000","last_undegraded":"2024-09-16T02:29:29.535134+0000","last_fullsized":"2024-09-16T02:29:29.535134+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640287,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640286,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672990,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:23.330 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:23.330 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:23.330 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:23.330 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:23.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:23.677 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:23.678 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":143,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:23.694 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:23.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:23.980 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:23.981 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:23.997 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-16T02:31:23.428284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:26.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:27.306 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:27.307 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:27.325 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-16T02:31:25.428803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.550311+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.550311+0000","last_peered":"2024-09-16T02:29:29.550311+0000","last_clean":"2024-09-16T02:29:29.550311+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.550311+0000","last_undegraded":"2024-09-16T02:29:29.550311+0000","last_fullsized":"2024-09-16T02:29:29.550311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554541+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.554541+0000","last_peered":"2024-09-16T02:29:29.554541+0000","last_clean":"2024-09-16T02:29:29.554541+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.554541+0000","last_undegraded":"2024-09-16T02:29:29.554541+0000","last_fullsized":"2024-09-16T02:29:29.554541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:27.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:27.624 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:27.625 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:27.641 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-16T02:31:27.429310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672992,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:27.642 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:27.642 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:27.642 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:27.642 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:27.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:27.984 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:27.984 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":147,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:28.000 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:28.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:28.296 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:28.297 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:28.314 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-16T02:31:27.429310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.554610+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.554610+0000","last_peered":"2024-09-16T02:29:29.554610+0000","last_clean":"2024-09-16T02:29:29.554610+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.554610+0000","last_undegraded":"2024-09-16T02:29:29.554610+0000","last_fullsized":"2024-09-16T02:29:29.554610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640288,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640287,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672992,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:31.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:31.629 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:31.629 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:31.645 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-16T02:31:31.430425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003066"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640289,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640288,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672992,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:31.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:31.947 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:31.948 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:31.965 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-16T02:31:31.430425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003066"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640289,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640288,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672992,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:31.966 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:31.966 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.547251+0000', 'last_change': '2024-09-16T02:29:29.548560+0000', 'last_active': '2024-09-16T02:29:29.547251+0000', 'last_peered': '2024-09-16T02:29:29.547251+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.547251+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.547251+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:31.966 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:31.966 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:31.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:32.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:32.303 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":151,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:32.320 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:32.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:32.605 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:32.606 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:32.622 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-16T02:31:31.430425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003066"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.547251+0000","last_change":"2024-09-16T02:29:29.548560+0000","last_active":"2024-09-16T02:29:29.547251+0000","last_peered":"2024-09-16T02:29:29.547251+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.547251+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.547251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.546491+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.546491+0000","last_peered":"2024-09-16T02:29:29.546491+0000","last_clean":"2024-09-16T02:29:29.546491+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.546491+0000","last_undegraded":"2024-09-16T02:29:29.546491+0000","last_fullsized":"2024-09-16T02:29:29.546491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640289,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640288,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672992,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:35.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:35.933 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:35.934 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:35.951 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T02:31:35.431592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640290,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640289,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672993,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:35.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:36.245 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:36.245 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T02:31:35.431592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640290,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640289,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672993,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:36.264 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:36.264 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:36.264 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:36.264 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 10, 'reported_epoch': 19, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-16T02:29:29.556434+0000', 'last_change': '2024-09-16T02:29:29.557816+0000', 'last_active': '2024-09-16T02:29:29.556434+0000', 'last_peered': '2024-09-16T02:29:29.556434+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.556434+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.556434+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:36.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:36.622 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":156,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":1},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:36.639 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:36.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:36.924 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:36.941 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-16T02:31:35.431592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":10,"reported_epoch":19,"state":"active+recovering+degraded","last_fresh":"2024-09-16T02:29:29.556434+0000","last_change":"2024-09-16T02:29:29.557816+0000","last_active":"2024-09-16T02:29:29.556434+0000","last_peered":"2024-09-16T02:29:29.556434+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.556434+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.556434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640290,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640289,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672993,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:39.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:40.243 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:40.243 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:40.259 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-16T02:31:39.432639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640290,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:40.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:40.551 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:40.552 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:40.568 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-16T02:31:39.432639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640290,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:40.568 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:40.568 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:40.569 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:40.569 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:40.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:40.906 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:40.906 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":160,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:40.923 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:40.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:41.211 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:41.211 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:41.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-16T02:31:39.432639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.549957+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.549957+0000","last_peered":"2024-09-16T02:29:29.549957+0000","last_clean":"2024-09-16T02:29:29.549957+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.549957+0000","last_undegraded":"2024-09-16T02:29:29.549957+0000","last_fullsized":"2024-09-16T02:29:29.549957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640290,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:44.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:44.539 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:44.540 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:44.558 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-16T02:31:43.433637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640291,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672995,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:44.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:44.858 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:44.858 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:44.874 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-16T02:31:43.433637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640291,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672995,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:44.875 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:44.875 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:44.875 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:44.875 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:44.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:45.217 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:45.217 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":164,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:45.234 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:45.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:45.522 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:45.540 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-16T02:31:45.434094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640291,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672995,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:48.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:48.851 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:48.851 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:48.868 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-16T02:31:47.434453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002861"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640291,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:48.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:49.158 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:49.159 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:49.175 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-16T02:31:47.434453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002861"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640291,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:49.176 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:49.176 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:49.176 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:49.176 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:49.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:49.535 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:49.535 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":169,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:49.552 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:49.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:49.847 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:49.848 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:49.865 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-16T02:31:49.434941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002819"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640292,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:52.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:53.176 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:53.177 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:53.194 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-16T02:31:51.435381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002742"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640292,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:53.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:53.485 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:53.485 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:53.503 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-16T02:31:53.435844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002706"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640293,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:53.504 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:53.504 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:53.504 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:53.505 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:53.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:53.855 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:53.856 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":173,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:53.872 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:53.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:54.153 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:54.153 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:54.169 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-16T02:31:53.435844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002706"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640293,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:57.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:57.482 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:57.482 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:57.500 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-16T02:31:57.436889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640293,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:57.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:57.794 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:57.795 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:57.810 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-16T02:31:57.436889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640293,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:31:57.811 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:31:57.811 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:31:57.811 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:31:57.811 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:31:57.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:31:58.140 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:58.141 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":177,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:31:58.158 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:31:58.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:31:58.457 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:31:58.457 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:31:58.474 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-16T02:31:57.436889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640293,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:01.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:01.794 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:01.794 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:01.812 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T02:32:01.437809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002867"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:01.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:02.110 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:02.110 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:02.127 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T02:32:01.437809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002867"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:02.127 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:02.127 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:02.128 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:02.128 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:02.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:02.485 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":182,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:02.503 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:02.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:02.784 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:02.784 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:02.800 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T02:32:01.437809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002867"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:05.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:06.108 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:06.108 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:06.125 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-16T02:32:05.438689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640295,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:06.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:06.400 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:06.400 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:06.416 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-16T02:32:05.438689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640295,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:06.417 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:06.417 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:06.417 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:06.417 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:06.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:06.759 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:06.759 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":186,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:06.775 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:06.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:07.060 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:07.061 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:07.077 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-16T02:32:05.438689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640295,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949672999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:10.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:10.388 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:10.388 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:10.406 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-16T02:32:09.439610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002720"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640296,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:10.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:10.699 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:10.699 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:10.715 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-16T02:32:09.439610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002720"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640296,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:10.716 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:10.716 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:10.716 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:10.716 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:10.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:11.044 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:11.044 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":190,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:11.061 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:11.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:11.345 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:11.345 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:11.361 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-16T02:32:09.439610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002720"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640296,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:14.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:14.676 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:14.677 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:14.694 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-16T02:32:13.440508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002699"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:14.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:14.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:14.996 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:15.013 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-16T02:32:13.440508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002699"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:15.014 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:15.014 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:15.014 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:15.014 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:15.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:15.371 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:15.371 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":194,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:15.388 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:15.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:15.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:15.688 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:15.706 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-16T02:32:15.440942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002696"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:18.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:19.013 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-16T02:32:17.441321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002632"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:19.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:19.321 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:19.321 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:19.337 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-16T02:32:17.441321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002632"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640297,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:19.338 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:19.338 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:19.338 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:19.338 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:19.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:19.687 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:19.688 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":199,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:19.705 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:19.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:19.988 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:19.989 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:20.005 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-16T02:32:19.441837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002669"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640298,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:23.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:23.318 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:23.334 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-16T02:32:21.442394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002784"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640298,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:23.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:23.633 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:23.633 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:23.650 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-16T02:32:23.442844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:23.651 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:23.651 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:23.652 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:23.652 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:23.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:23.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:23.996 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:24.012 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:24.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:24.292 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:24.293 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:24.308 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-16T02:32:23.442844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002795"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:27.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:27.618 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:27.619 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:27.635 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-16T02:32:27.443687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:27.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:27.913 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:27.913 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:27.929 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-16T02:32:27.443687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:27.929 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:27.930 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:27.930 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:27.930 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:27.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:28.273 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:28.273 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":207,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:28.292 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:28.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:28.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:28.579 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:28.594 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-16T02:32:27.443687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:31.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:31.899 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:31.900 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:31.915 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-16T02:32:31.444602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002765"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640300,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:31.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:32.201 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-16T02:32:31.444602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002765"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640300,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:32.219 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:32.219 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:32.219 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:32.219 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:32.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:32.573 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:32.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":212,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:32.590 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:32.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:32.868 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:32.868 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:32.886 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-16T02:32:31.444602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002765"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640300,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:35.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:36.197 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:36.198 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:36.215 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-16T02:32:35.445534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002690"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640301,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:36.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:36.504 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:36.504 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:36.520 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-16T02:32:35.445534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002690"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640301,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:36.521 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:36.521 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:36.521 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:36.521 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:36.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:36.852 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:36.852 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:36.868 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:36.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:37.138 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:37.138 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:37.154 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-16T02:32:35.445534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002690"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640301,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:40.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:40.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:40.468 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:40.486 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-16T02:32:39.446540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002853"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640302,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:40.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:40.788 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:40.789 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:40.806 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-16T02:32:39.446540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002853"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640302,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:40.807 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:40.807 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:40.807 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:40.807 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:40.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:41.156 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:41.156 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:41.173 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:41.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:41.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:41.468 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:41.485 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-16T02:32:41.447028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002955"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640302,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:44.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:44.793 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:44.794 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:44.810 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-16T02:32:43.447491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002889"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640303,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:44.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:45.105 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:45.106 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:45.122 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-16T02:32:43.447491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002889"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640303,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:45.122 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:45.123 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:45.123 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:45.123 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:45.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:45.451 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":225,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:45.468 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:45.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:45.770 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:45.785 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-16T02:32:45.447967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002929"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640303,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:48.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:49.087 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:49.087 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:49.103 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-16T02:32:47.448481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002946"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640303,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:49.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:49.401 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:49.419 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-16T02:32:47.448481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002946"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640303,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:49.420 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:49.420 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:49.420 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:49.420 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:49.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:49.783 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:49.783 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":229,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:49.801 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:49.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:50.086 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:50.102 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-16T02:32:49.448940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002896"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640305,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640304,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:53.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:53.411 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:53.411 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:53.428 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-16T02:32:51.449314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640305,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640304,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:53.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:53.701 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:53.701 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:53.717 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-16T02:32:53.449763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002735"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:53.718 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:53.718 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:53.718 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:53.718 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:53.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:54.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:54.051 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":233,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:54.067 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:54.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:54.350 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:54.351 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:54.367 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-16T02:32:53.449763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002735"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:57.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:57.676 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:57.677 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:57.693 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-16T02:32:57.450730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002763"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:57.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:57.988 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:57.988 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:58.006 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-16T02:32:57.450730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002763"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:32:58.007 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:32:58.007 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:32:58.007 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:32:58.007 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:32:58.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:32:58.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:58.377 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":237,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:32:58.396 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:32:58.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:32:58.677 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:32:58.677 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:32:58.694 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-16T02:32:57.450730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002763"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640306,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:01.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:02.016 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:02.032 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-16T02:33:01.451711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640306,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:02.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:02.328 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:02.346 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-16T02:33:01.451711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640306,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:02.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:02.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:02.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:02.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:02.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:02.697 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:02.697 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":242,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:02.714 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:02.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:02.987 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:02.987 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:03.004 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-16T02:33:01.451711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640306,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:06.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:06.314 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:06.331 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-16T02:33:05.452620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640307,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:06.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:06.624 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:06.624 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:06.640 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-16T02:33:05.452620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640307,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:06.641 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:06.641 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:06.641 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:06.641 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:06.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:06.976 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":246,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:06.993 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:06.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:07.282 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:07.282 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:07.298 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-16T02:33:05.452620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640307,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:10.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:10.605 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:10.605 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:10.621 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-16T02:33:09.453588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002858"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640308,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:10.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:10.893 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:10.893 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:10.908 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-16T02:33:09.453588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002858"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640308,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:10.909 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:10.909 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:10.909 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:10.909 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:10.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:11.248 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:11.249 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":250,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:11.265 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:11.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:11.557 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:11.558 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:11.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-16T02:33:11.454027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002844"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640308,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:14.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:14.885 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:14.885 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:14.903 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-16T02:33:13.454425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002714"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:14.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:15.197 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:15.197 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:15.213 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-16T02:33:13.454425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002714"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:15.214 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:15.214 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:15.214 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:15.214 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:15.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:15.552 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:15.552 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":255,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:15.569 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:15.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:15.859 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:15.875 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-16T02:33:15.454914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002740"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:18.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:19.179 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:19.195 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-16T02:33:17.455347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002726"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:19.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:19.471 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:19.472 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:19.489 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-16T02:33:19.455828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002763"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:19.490 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:19.490 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:19.491 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:19.491 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:19.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:19.833 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:19.833 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":259,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:19.850 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:19.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:20.131 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:20.131 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:20.146 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-16T02:33:19.455828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002763"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:23.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:23.456 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:23.457 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:23.475 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-16T02:33:21.456431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002843"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640311,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:23.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:23.760 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:23.761 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:23.777 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-16T02:33:23.456976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002948"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:23.778 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:23.778 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:23.779 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:23.779 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:23.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:24.121 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:24.121 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":263,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:24.138 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:24.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:24.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:24.420 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:24.436 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-16T02:33:23.456976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002948"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:27.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:27.744 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:27.745 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:27.762 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T02:33:27.458074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003160"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:27.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:28.062 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:28.062 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:28.080 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T02:33:27.458074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003160"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:28.080 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:28.080 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:28.081 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:28.081 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:28.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:28.420 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":268,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:28.436 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:28.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:28.714 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:28.714 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:28.730 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T02:33:27.458074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003160"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.536069+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:29:29.536069+0000","last_peered":"2024-09-16T02:29:29.536069+0000","last_clean":"2024-09-16T02:29:29.536069+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:29:29.536069+0000","last_undegraded":"2024-09-16T02:29:29.536069+0000","last_fullsized":"2024-09-16T02:29:29.536069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555743+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:29:29.555743+0000","last_peered":"2024-09-16T02:29:29.555743+0000","last_clean":"2024-09-16T02:29:29.555743+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:29:29.555743+0000","last_undegraded":"2024-09-16T02:29:29.555743+0000","last_fullsized":"2024-09-16T02:29:29.555743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560088+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:29:29.560088+0000","last_peered":"2024-09-16T02:29:29.560088+0000","last_clean":"2024-09-16T02:29:29.560088+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:29:29.560088+0000","last_undegraded":"2024-09-16T02:29:29.560088+0000","last_fullsized":"2024-09-16T02:29:29.560088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640312,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:31.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:32.040 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:32.040 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:32.059 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-16T02:33:31.459048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003219"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640313,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:32.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:32.360 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:32.360 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:32.377 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-16T02:33:31.459048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003219"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640313,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:32.378 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:32.378 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:32.379 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:32.379 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:32.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:32.734 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:32.734 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":272,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:32.756 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:32.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:33.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:33.045 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:33.061 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-16T02:33:31.459048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003219"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.551659+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:29:29.551659+0000","last_peered":"2024-09-16T02:29:29.551659+0000","last_clean":"2024-09-16T02:29:29.551659+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:29:29.551659+0000","last_undegraded":"2024-09-16T02:29:29.551659+0000","last_fullsized":"2024-09-16T02:29:29.551659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640313,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:36.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:36.368 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:36.368 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:36.384 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-16T02:33:35.460119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003143"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:36.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:36.658 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:36.675 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-16T02:33:35.460119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003143"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:36.676 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:36.676 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.548560+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:29:29.548560+0000', 'last_peered': '2024-09-16T02:29:29.548560+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:29:29.548560+0000', 'last_undegraded': '2024-09-16T02:29:29.535725+0000', 'last_fullsized': '2024-09-16T02:29:29.548560+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:36.676 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:36.676 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:36.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:37.037 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:37.037 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":276,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:37.056 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:37.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:37.339 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:37.340 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:37.356 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-16T02:33:35.460119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003143"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.548560+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:29:29.548560+0000","last_peered":"2024-09-16T02:29:29.548560+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:29:29.548560+0000","last_undegraded":"2024-09-16T02:29:29.535725+0000","last_fullsized":"2024-09-16T02:29:29.548560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:40.357 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:40.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:40.673 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:40.691 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-16T02:33:39.461085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003011"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640314,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:40.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:40.991 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:40.991 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:41.009 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-16T02:33:39.461085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003011"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.560090+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:29:29.560090+0000","last_peered":"2024-09-16T02:29:29.560090+0000","last_clean":"2024-09-16T02:29:29.560090+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:29:29.560090+0000","last_undegraded":"2024-09-16T02:29:29.560090+0000","last_fullsized":"2024-09-16T02:29:29.560090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":11,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:29:29.557816+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:29:29.557816+0000","last_peered":"2024-09-16T02:29:29.557816+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:29:29.557816+0000","last_undegraded":"2024-09-16T02:29:29.540498+0000","last_fullsized":"2024-09-16T02:29:29.557816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640314,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:41.009 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:41.010 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:41.010 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:41.010 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 11, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:29:29.557816+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:29:29.557816+0000', 'last_peered': '2024-09-16T02:29:29.557816+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:29:29.557816+0000', 'last_undegraded': '2024-09-16T02:29:29.540498+0000', 'last_fullsized': '2024-09-16T02:29:29.557816+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:41.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:41.378 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:41.378 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":280,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:41.397 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:41.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:41.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:41.690 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:41.705 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-16T02:33:41.461541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003040"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640314,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:44.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:45.009 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:45.010 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:45.025 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-16T02:33:43.461954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640316,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:45.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:45.311 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:45.327 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-16T02:33:43.461954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640316,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:45.328 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:45.328 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:45.328 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:45.328 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:45.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:45.673 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:45.673 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":285,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:45.689 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:45.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:45.985 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:45.985 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:46.002 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-16T02:33:45.462404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002799"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640316,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:49.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:49.312 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:49.329 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-16T02:33:47.462750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640316,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:49.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:49.622 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:49.622 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:49.640 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-16T02:33:49.463178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002585"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640317,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640316,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:49.640 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:49.640 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:49.641 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:49.641 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:49.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:49.986 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:49.986 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":289,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:50.003 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:50.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:50.292 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:50.293 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:50.308 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-16T02:33:49.463178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002585"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:29:29.555694+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:29:29.555694+0000","last_peered":"2024-09-16T02:29:29.555694+0000","last_clean":"2024-09-16T02:29:29.555694+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:29:29.555694+0000","last_undegraded":"2024-09-16T02:29:29.555694+0000","last_fullsized":"2024-09-16T02:29:29.555694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640317,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640316,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:53.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:53.615 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:53.615 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:53.633 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-16T02:33:53.463989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002448"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:53.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:53.922 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:53.922 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:53.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-16T02:33:53.463989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002448"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:53.939 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:53.939 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:53.940 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:53.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:53.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:54.295 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:54.295 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":293,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:54.312 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:54.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:54.596 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:54.596 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:54.612 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-16T02:33:53.463989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002448"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:57.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:57.923 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:57.924 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:57.941 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-16T02:33:57.464753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002349"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:57.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:58.236 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:58.236 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:58.254 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-16T02:33:57.464753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002349"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:33:58.254 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:33:58.254 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:33:58.255 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:33:58.255 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:33:58.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:33:58.613 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:58.613 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":298,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:33:58.630 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:33:58.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:33:58.902 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:33:58.902 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:33:58.918 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-16T02:33:57.464753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002349"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:01.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:02.227 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:02.245 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-16T02:34:01.465777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002598"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640319,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640318,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:02.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:02.536 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:02.537 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:02.553 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-16T02:34:01.465777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002598"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640319,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640318,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:02.554 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:02.554 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:02.554 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:02.554 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:02.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:02.918 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:02.918 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":302,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:02.937 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:02.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:03.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:03.211 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:03.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-16T02:34:01.465777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002598"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640319,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640318,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:06.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:06.539 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:06.540 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:06.557 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-16T02:34:05.466780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:06.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:06.863 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:06.863 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:06.881 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-16T02:34:05.466780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:06.882 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:06.882 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:06.883 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:06.883 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:06.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:07.226 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:07.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":306,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:07.243 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:07.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:07.520 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:07.520 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:07.536 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-16T02:34:07.467229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002958"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:10.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:10.851 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:10.851 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:10.869 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-16T02:34:09.467679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002925"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640320,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:10.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:11.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:11.165 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:11.181 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-16T02:34:09.467679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002925"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640320,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:11.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:11.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:11.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:11.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:11.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:11.531 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:11.531 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":311,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:11.548 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:11.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:11.831 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:11.831 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:11.847 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T02:34:11.468178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640320,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:14.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:15.159 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:15.159 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:15.176 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-16T02:34:13.468651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002874"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640322,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640321,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:15.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:15.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:15.468 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:15.484 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-16T02:34:13.468651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002874"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640322,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640321,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:15.484 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:15.484 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:15.485 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:15.485 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:15.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:15.819 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:15.820 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":315,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:15.836 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:15.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:16.114 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:16.114 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:16.130 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-16T02:34:15.469127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002834"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640322,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640321,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:19.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:19.447 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:19.448 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:19.466 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-16T02:34:17.469475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640322,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640321,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:19.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:19.764 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:19.764 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:19.782 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-16T02:34:19.469932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640322,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:19.782 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:19.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:19.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:19.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:19.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:20.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:20.127 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":319,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:20.143 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:20.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:20.417 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:20.418 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:20.434 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-16T02:34:19.469932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002702"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640322,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:23.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:23.751 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:23.751 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:23.769 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-16T02:34:23.470810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002631"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:23.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:24.061 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:24.062 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:24.079 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-16T02:34:23.470810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002631"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:24.080 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:24.080 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:24.080 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:24.080 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:24.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:24.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:24.420 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":324,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:24.437 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:24.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:24.727 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:24.727 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:24.742 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-16T02:34:23.470810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002631"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:27.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:28.056 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:28.056 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:28.073 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-16T02:34:27.471740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002613"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:28.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:28.366 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:28.383 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-16T02:34:27.471740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002613"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:28.383 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:28.384 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:28.384 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:28.384 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:28.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:28.738 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:28.738 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":328,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:28.754 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:28.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:29.054 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:29.071 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-16T02:34:27.471740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002613"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:32.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:32.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:32.378 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:32.394 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-16T02:34:31.472817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640324,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:32.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:32.668 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:32.669 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:32.685 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-16T02:34:31.472817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640324,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:32.686 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:32.686 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:32.686 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:32.686 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:32.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:33.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:33.051 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":332,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:33.069 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:33.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:33.364 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:33.365 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:33.381 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-16T02:34:31.472817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640324,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:36.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:36.690 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:36.706 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-16T02:34:35.473781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640325,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:36.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:36.981 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:36.997 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-16T02:34:35.473781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640325,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:36.998 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:36.998 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:36.998 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:36.998 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:36.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:37.354 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:37.354 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":336,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:37.372 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:37.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:37.673 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:37.674 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:37.690 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-16T02:34:37.474203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002956"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640325,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:40.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:40.998 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:40.998 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:41.016 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-16T02:34:39.474543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002802"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640327,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640326,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:41.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:41.300 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:41.300 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:41.316 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-16T02:34:39.474543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002802"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640327,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640326,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:41.317 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:41.317 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:41.317 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:41.317 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:41.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:41.652 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:41.652 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:41.669 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:41.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:41.947 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:41.948 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:41.965 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-16T02:34:41.475033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002738"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640327,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640326,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:44.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:45.290 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T02:34:43.475442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002624"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640327,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:45.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:45.583 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:45.583 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:45.598 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-16T02:34:45.475927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640327,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:45.599 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:45.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:45.600 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:45.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:45.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:45.949 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:45.949 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":345,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:45.966 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:45.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:46.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:46.263 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:46.280 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-16T02:34:45.475927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640327,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:49.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:49.597 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:49.597 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:49.615 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-16T02:34:49.476878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002675"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640329,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640328,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:49.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:49.907 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:49.907 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:49.925 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-16T02:34:49.476878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002675"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640329,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640328,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:49.926 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:49.926 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:49.926 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:49.927 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:49.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:50.289 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:50.289 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":349,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:50.306 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:50.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:50.605 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:50.623 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-16T02:34:49.476878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002675"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640329,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640328,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:53.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:53.927 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:53.927 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:53.943 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-16T02:34:53.477896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640329,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:53.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:54.227 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:54.246 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-16T02:34:53.477896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640329,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:54.246 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:54.247 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:54.247 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:54.247 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:54.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:54.587 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:54.587 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":354,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:54.605 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:54.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:54.878 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-16T02:34:53.477896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640329,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:57.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:58.197 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:58.197 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:58.213 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-16T02:34:57.478899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640329,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:58.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:58.486 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:58.487 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:58.503 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-16T02:34:57.478899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640329,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:34:58.504 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:34:58.504 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:34:58.504 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:34:58.504 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:34:58.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:34:58.866 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:58.866 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":358,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:34:58.885 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:34:58.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:34:59.181 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:34:59.181 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:34:59.199 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-09-16T02:34:57.478899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002971"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640329,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:02.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:02.503 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:02.519 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T02:35:01.479957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003078"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640331,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640330,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:02.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:02.812 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:02.812 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:02.828 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T02:35:01.479957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003078"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640331,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640330,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:02.828 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:02.829 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:02.829 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:02.829 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:02.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:03.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:03.167 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":362,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:03.183 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:03.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:03.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:03.468 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:03.484 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T02:35:01.479957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003078"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640331,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640330,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:06.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:06.802 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:06.802 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:06.819 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-16T02:35:05.480783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640331,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:06.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:07.109 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:07.110 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:07.128 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-16T02:35:05.480783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640331,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:07.128 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:07.128 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:07.129 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:07.129 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:07.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:07.491 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:07.491 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":367,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:07.509 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:07.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:07.801 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:07.801 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-16T02:35:07.481226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002799"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640331,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:10.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:11.123 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:11.124 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:11.139 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-16T02:35:09.481735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002835"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640332,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:11.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:11.423 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:11.439 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-16T02:35:09.481735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002835"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640332,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:11.440 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:11.440 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:11.440 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:11.440 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:11.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:11.788 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:11.788 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":371,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:11.805 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:11.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:12.075 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:12.075 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:12.091 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T02:35:11.482190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002756"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640332,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:15.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:15.405 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:15.405 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:15.422 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-16T02:35:13.482545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002588"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640334,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640334,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:15.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:15.706 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:15.707 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:15.723 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-16T02:35:15.483011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640334,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640334,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:15.724 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:15.724 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:15.724 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:15.724 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:15.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:16.079 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:16.079 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":375,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:16.095 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:16.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:16.395 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:16.395 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:16.412 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-16T02:35:15.483011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640334,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640334,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:19.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:19.723 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:19.724 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:19.741 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-16T02:35:19.483798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002572"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640335,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:19.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:20.023 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:20.039 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-16T02:35:19.483798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002572"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640335,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:20.039 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:20.039 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:20.040 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:20.040 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:20.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:20.392 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:20.392 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":379,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:20.409 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:20.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:20.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:20.694 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:20.711 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-16T02:35:19.483798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002572"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640335,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:23.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:24.024 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:24.024 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:24.042 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T02:35:23.484818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640336,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:24.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:24.331 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:24.332 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:24.347 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T02:35:23.484818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640336,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:24.348 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:24.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:24.348 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:24.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:24.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:24.714 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:24.714 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":384,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:24.732 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:24.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:25.025 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:25.026 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:25.043 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T02:35:23.484818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640336,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:28.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:28.356 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:28.357 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:28.374 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T02:35:27.485850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002839"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640336,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:28.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:28.673 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:28.673 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:28.689 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T02:35:27.485850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002839"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640336,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:28.690 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:28.690 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:28.690 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:28.691 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:28.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:29.032 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:29.032 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":388,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:29.049 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:29.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:29.319 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:29.335 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T02:35:27.485850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002839"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640336,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:32.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:32.649 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:32.649 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:32.667 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-16T02:35:31.486903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003104"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640337,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:32.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:32.959 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:32.960 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:32.977 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-16T02:35:31.486903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003104"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:22.490825+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:31:22.490825+0000","last_peered":"2024-09-16T02:31:22.490825+0000","last_clean":"2024-09-16T02:31:22.490825+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:31:22.490825+0000","last_undegraded":"2024-09-16T02:31:22.490825+0000","last_fullsized":"2024-09-16T02:31:22.490825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:32.491920+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:31:32.491920+0000","last_peered":"2024-09-16T02:31:32.491920+0000","last_clean":"2024-09-16T02:31:32.491920+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:31:32.491920+0000","last_undegraded":"2024-09-16T02:31:32.491920+0000","last_fullsized":"2024-09-16T02:31:32.491920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640337,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:32.978 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:32.978 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:32.978 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:32.978 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:32.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:33.325 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:33.325 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":392,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:33.342 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:33.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:33.618 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:33.618 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:33.634 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-16T02:35:33.487334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003101"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640338,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640338,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:36.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:36.947 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:36.965 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-16T02:35:35.487608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640338,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640338,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:36.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:37.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:37.263 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:37.278 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-16T02:35:35.487608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488054+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:31:25.488054+0000","last_peered":"2024-09-16T02:31:25.488054+0000","last_clean":"2024-09-16T02:31:25.488054+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:31:25.488054+0000","last_undegraded":"2024-09-16T02:31:25.488054+0000","last_fullsized":"2024-09-16T02:31:25.488054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:25.488007+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:31:25.488007+0000","last_peered":"2024-09-16T02:31:25.488007+0000","last_clean":"2024-09-16T02:31:25.488007+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:31:25.488007+0000","last_undegraded":"2024-09-16T02:31:25.488007+0000","last_fullsized":"2024-09-16T02:31:25.488007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640338,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640338,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:37.279 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:37.279 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:37.279 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:37.279 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:37.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:37.623 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:37.623 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":397,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:37.640 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:37.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:37.931 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:37.975 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:37.975 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T02:35:37.488064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002714"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640338,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640338,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:40.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:41.285 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:41.286 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:41.302 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-16T02:35:39.488549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002699"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:41.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:41.596 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:41.596 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:41.614 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-16T02:35:41.489035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002647"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:41.615 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:41.615 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:32.491975+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:31:32.491975+0000', 'last_peered': '2024-09-16T02:31:32.491975+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:31:32.491975+0000', 'last_undegraded': '2024-09-16T02:31:32.491975+0000', 'last_fullsized': '2024-09-16T02:31:32.491975+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:41.615 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:41.615 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:41.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:41.953 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:41.953 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":401,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:41.970 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:41.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:42.267 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:42.267 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:42.285 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-16T02:35:41.489035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002647"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:32.491975+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:31:32.491975+0000","last_peered":"2024-09-16T02:31:32.491975+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:31:32.491975+0000","last_undegraded":"2024-09-16T02:31:32.491975+0000","last_fullsized":"2024-09-16T02:31:32.491975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640339,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640339,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:45.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:45.599 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:45.600 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:45.618 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-16T02:35:45.490144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002809"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640340,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:45.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:45.897 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:45.897 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:45.914 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-16T02:35:45.490144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002809"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640340,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:45.915 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:45.915 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:45.915 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:45.915 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 12, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:31:35.490032+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:31:35.490032+0000', 'last_peered': '2024-09-16T02:31:35.490032+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:31:35.490032+0000', 'last_undegraded': '2024-09-16T02:31:35.490032+0000', 'last_fullsized': '2024-09-16T02:31:35.490032+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:45.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:46.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:46.258 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":405,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:46.275 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:46.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:46.566 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:46.566 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:46.583 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-16T02:35:45.490144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002809"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":12,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:31:35.490032+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:31:35.490032+0000","last_peered":"2024-09-16T02:31:35.490032+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:31:35.490032+0000","last_undegraded":"2024-09-16T02:31:35.490032+0000","last_fullsized":"2024-09-16T02:31:35.490032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640340,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:49.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:49.896 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:49.896 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:49.914 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-16T02:35:49.491149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640341,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640341,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:49.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:50.205 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:50.205 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:50.221 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-16T02:35:49.491149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640341,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640341,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:50.222 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:50.222 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:50.223 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:50.223 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:50.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:50.562 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:50.563 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":410,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:50.579 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:50.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:50.877 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:50.878 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-16T02:35:49.491149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:30.489000+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:31:30.489000+0000","last_peered":"2024-09-16T02:31:30.489000+0000","last_clean":"2024-09-16T02:31:30.489000+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:31:30.489000+0000","last_undegraded":"2024-09-16T02:31:30.489000+0000","last_fullsized":"2024-09-16T02:31:30.489000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640341,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640341,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:53.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:54.207 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:54.207 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:54.226 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-16T02:35:53.492179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640342,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:54.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:54.520 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:54.521 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:54.538 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-16T02:35:53.492179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640342,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:54.539 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:54.539 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:54.540 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:54.540 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:54.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:54.875 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":414,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:54.892 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:54.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:55.162 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:55.162 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:55.178 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-16T02:35:53.492179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003144"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640342,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:58.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:58.488 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:58.488 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:58.504 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-16T02:35:57.493236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003092"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640342,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:58.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:58.791 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:58.791 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:58.799 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-16T02:35:57.493236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003092"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640342,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:35:58.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:35:58.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:35:58.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:35:58.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:35:58.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:35:59.183 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:59.183 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":418,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:35:59.202 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:35:59.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:35:59.496 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:35:59.497 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:35:59.512 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-16T02:35:59.493715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:31:40.490914+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:31:40.490914+0000","last_peered":"2024-09-16T02:31:40.490914+0000","last_clean":"2024-09-16T02:31:40.490914+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:31:40.490914+0000","last_undegraded":"2024-09-16T02:31:40.490914+0000","last_fullsized":"2024-09-16T02:31:40.490914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640343,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640343,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:02.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:02.817 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:02.817 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:02.833 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T02:36:01.494202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003052"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640343,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640343,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:02.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:03.124 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:03.125 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:03.141 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T02:36:01.494202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003052"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640343,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640343,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:03.142 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:03.142 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:03.142 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:03.142 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:03.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:03.481 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:03.481 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":423,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:03.498 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:03.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:03.774 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:03.774 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:03.789 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-16T02:36:03.494638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002982"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640344,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:06.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:07.103 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:07.103 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:07.120 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-16T02:36:05.494978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002798"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640344,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:07.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:07.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:07.417 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:07.435 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-16T02:36:05.494978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002798"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640344,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:07.436 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:07.436 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:07.436 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:07.436 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:07.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:07.791 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":427,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:07.808 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:07.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:08.087 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:08.087 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:08.104 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-16T02:36:07.495389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002594"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640344,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640344,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:11.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:11.413 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:11.431 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-16T02:36:09.495831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002595"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640345,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640345,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:11.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:11.715 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:11.732 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-16T02:36:11.496307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002592"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640345,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640345,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:11.733 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:11.733 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:11.733 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:11.733 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:11.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:12.098 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:12.098 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":431,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T02:30:13.410098+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:12.116 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:12.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:12.410 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:12.411 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:12.428 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-16T02:36:11.496307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002592"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640345,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640345,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:15.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:15.744 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:15.744 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:15.762 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T02:36:15.497252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640346,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640346,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:15.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:16.061 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:16.061 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:16.078 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T02:36:15.497252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640346,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640346,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:16.079 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:16.079 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:16.079 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:16.079 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:16.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:16.410 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:16.411 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":436,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:16.427 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:16.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:16.710 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:16.711 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:16.726 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T02:36:15.497252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640346,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640346,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:19.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:20.037 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:20.054 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-16T02:36:19.498286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640347,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:20.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:20.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:20.330 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:20.346 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-16T02:36:19.498286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640347,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:20.346 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:20.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:20.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:20.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:20.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:20.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:20.688 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":440,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:20.704 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:20.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:20.987 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:20.987 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:21.004 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-16T02:36:19.498286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640347,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:24.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:24.310 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:24.310 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:24.328 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-16T02:36:23.499308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640348,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:24.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:24.623 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:24.624 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:24.642 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-16T02:36:23.499308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640348,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:24.642 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:24.642 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:24.643 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:24.643 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:24.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:25.007 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":444,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:25.025 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:25.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:25.323 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:25.323 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:25.339 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-16T02:36:23.499308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640348,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:28.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:28.657 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:28.657 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:28.675 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-16T02:36:27.500303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003050"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640348,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:28.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:28.971 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:28.972 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:28.988 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-16T02:36:27.500303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003050"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640348,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640348,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:28.989 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:28.989 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:28.989 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:28.989 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:28.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:29.321 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":448,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:29.338 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:29.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:29.613 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:29.629 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-16T02:36:29.500734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002955"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640349,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640349,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:32.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:32.938 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:32.938 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:32.954 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-16T02:36:31.501168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002882"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640349,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640349,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:32.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:33.251 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:33.251 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:33.268 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-16T02:36:31.501168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002882"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640349,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640349,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:33.269 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:33.269 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:33.269 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:33.269 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:33.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:33.618 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:33.618 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":453,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:33.634 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:33.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:33.908 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:33.908 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:33.924 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-16T02:36:33.501765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002969"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640350,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:36.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:37.236 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:37.236 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:37.254 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-16T02:36:35.502388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003079"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640350,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:37.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:37.538 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:37.539 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:37.557 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-16T02:36:37.502869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640350,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:37.558 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:37.558 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:37.558 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:37.558 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:37.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:37.888 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:37.888 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":457,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:37.905 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:37.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:38.196 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:38.196 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:38.214 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-16T02:36:37.502869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640350,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:41.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:41.525 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:41.525 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:41.543 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T02:36:41.503943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640351,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640351,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:41.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:41.842 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:41.842 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:41.860 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T02:36:41.503943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640351,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640351,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:41.861 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:41.861 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:41.861 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:41.861 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:41.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:42.221 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:42.221 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":461,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:42.238 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:42.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:42.515 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:42.516 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:42.532 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T02:36:41.503943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640351,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640351,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:45.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:45.837 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:45.853 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-16T02:36:45.504966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640352,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640352,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:45.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:46.143 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:46.143 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:46.160 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-16T02:36:45.504966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640352,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640352,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:46.161 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:46.161 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:46.161 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:46.161 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:46.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:46.523 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:46.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:46.809 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:46.810 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:46.826 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-16T02:36:45.504966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640352,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640352,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:49.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:50.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:50.133 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:50.150 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-16T02:36:49.505996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003127"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640353,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:50.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:50.442 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:50.460 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-16T02:36:49.505996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003127"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640353,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:50.460 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:50.460 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:50.461 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:50.461 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:50.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:50.815 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:50.816 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":470,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:50.832 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:50.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:51.118 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:51.118 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:51.135 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-16T02:36:49.505996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003127"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640354,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640353,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:54.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:54.439 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:54.439 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:54.455 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-16T02:36:53.507076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003132"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640354,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:54.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:54.747 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:54.747 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:54.763 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-16T02:36:53.507076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003132"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640354,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:54.764 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:54.764 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:54.764 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:54.764 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:54.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:55.111 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":474,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:55.128 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:55.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:55.401 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:55.401 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:55.417 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-16T02:36:53.507076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003132"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640354,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:58.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:58.729 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:58.730 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:58.748 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T02:36:57.508003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003037"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640354,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:58.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:59.047 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:59.047 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:59.064 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T02:36:57.508003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003037"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640354,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:36:59.065 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:36:59.065 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:36:59.066 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:36:59.066 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:36:59.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:36:59.405 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:59.405 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":478,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:36:59.422 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:36:59.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:36:59.700 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:36:59.701 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:36:59.718 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-16T02:36:59.508496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003056"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640356,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640355,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:02.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:03.031 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:03.032 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:03.047 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-16T02:37:01.508877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640356,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640355,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:03.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:03.353 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:03.354 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:03.372 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-16T02:37:01.508877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640356,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640355,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:03.372 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:03.373 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:03.373 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:03.373 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:03.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:03.720 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:03.721 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":483,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:03.738 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:03.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:04.013 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:04.013 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:04.029 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-16T02:37:03.509332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002792"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640357,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640356,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:07.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:07.343 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:07.343 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:07.361 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-16T02:37:05.509688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640357,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640356,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:07.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:07.659 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:07.660 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:07.678 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-16T02:37:07.510075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002517"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640357,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640356,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:07.679 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:07.679 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:07.679 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:07.679 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:07.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:08.029 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:08.029 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":487,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:08.046 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:08.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:08.327 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:08.327 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:08.343 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-16T02:37:07.510075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002517"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640357,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640356,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:11.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:11.653 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:11.672 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-16T02:37:11.510980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002484"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640358,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640357,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:11.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:11.965 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:11.966 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:11.982 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-16T02:37:11.510980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002484"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640358,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640357,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:11.983 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:11.983 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:11.983 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:11.983 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:11.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:12.317 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":491,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:12.334 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:12.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:12.605 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:12.605 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:12.620 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-16T02:37:11.510980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002484"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640358,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640357,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:15.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:15.922 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:15.938 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-16T02:37:15.511855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002523"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640358,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:15.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:16.227 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:16.227 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:16.243 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-16T02:37:15.511855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002523"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640358,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:16.244 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:16.244 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:16.244 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:16.244 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:16.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:16.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:16.582 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":496,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:16.598 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:16.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:16.877 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:16.877 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:16.894 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-16T02:37:15.511855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002523"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640358,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:19.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:20.199 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:20.200 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:20.216 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-16T02:37:19.512859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002784"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640360,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640359,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:20.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:20.502 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:20.503 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:20.519 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-16T02:37:19.512859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002784"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640360,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640359,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:20.519 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:20.519 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:20.520 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:20.520 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:20.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:20.870 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:20.870 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":500,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:20.887 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:20.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:21.171 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:21.171 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:21.188 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-16T02:37:19.512859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002784"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640360,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640359,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:24.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:24.499 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:24.500 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:24.517 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-16T02:37:23.513915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640360,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:24.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:24.791 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:24.791 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:24.807 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-16T02:37:23.513915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640360,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:24.808 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:24.808 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:24.808 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:24.808 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:24.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:25.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:25.166 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:25.183 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:25.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:25.465 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:25.481 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-16T02:37:23.513915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640360,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:28.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:28.792 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:28.792 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:28.810 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-16T02:37:27.514772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640360,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:28.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:29.099 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:29.099 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:29.115 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-16T02:37:27.514772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640360,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:29.116 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:29.116 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:29.116 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:29.116 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:29.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:29.473 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:29.473 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":509,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:29.491 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:29.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:29.809 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:29.812 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:29.812 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-16T02:37:29.515338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640362,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640361,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:32.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:33.123 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:33.124 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:33.140 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-16T02:37:31.515664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002804"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640362,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640361,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:33.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:33.421 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:33.422 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:33.438 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-16T02:37:31.515664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002804"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:32.506769+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:33:32.506769+0000","last_peered":"2024-09-16T02:33:32.506769+0000","last_clean":"2024-09-16T02:33:32.506769+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:33:32.506769+0000","last_undegraded":"2024-09-16T02:33:32.506769+0000","last_fullsized":"2024-09-16T02:33:32.506769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640362,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640361,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:33.439 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:33.439 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:33.439 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:33.439 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:33.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:33.803 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:33.804 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":513,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:33.823 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:33.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:34.120 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:34.138 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-16T02:37:33.516125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002723"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640363,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640362,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:37.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:37.449 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:37.450 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:37.468 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-16T02:37:35.516532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002616"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640363,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640362,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:37.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:37.756 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:37.756 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:37.773 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-16T02:37:37.516965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002587"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640363,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640362,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:37.773 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:37.774 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:37.774 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:37.774 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:37.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:38.112 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:38.112 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":517,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:38.128 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:38.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:39.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:39.259 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:39.283 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-16T02:37:37.516965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002587"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:27.506406+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:33:27.506406+0000","last_peered":"2024-09-16T02:33:27.506406+0000","last_clean":"2024-09-16T02:33:27.506406+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:33:27.506406+0000","last_undegraded":"2024-09-16T02:33:27.506406+0000","last_fullsized":"2024-09-16T02:33:27.506406+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509991+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:33:30.509991+0000","last_peered":"2024-09-16T02:33:30.509991+0000","last_clean":"2024-09-16T02:33:30.509991+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:33:30.509991+0000","last_undegraded":"2024-09-16T02:33:30.509991+0000","last_fullsized":"2024-09-16T02:33:30.509991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:30.509941+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:33:30.509941+0000","last_peered":"2024-09-16T02:33:30.509941+0000","last_clean":"2024-09-16T02:33:30.509941+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:33:30.509941+0000","last_undegraded":"2024-09-16T02:33:30.509941+0000","last_fullsized":"2024-09-16T02:33:30.509941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640363,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640362,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:42.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:42.587 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:42.588 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:42.604 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-16T02:37:41.517979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002641"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640364,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640363,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:42.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:42.902 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:42.902 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:42.919 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-16T02:37:41.517979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002641"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640364,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640363,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:42.920 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:42.920 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:42.920 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:42.920 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:42.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:43.262 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":522,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:43.279 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:43.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:43.557 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:43.558 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:43.572 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-16T02:37:43.518490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002826"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640364,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:46.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:46.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:46.880 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:46.897 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-16T02:37:45.518980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002855"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640364,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:46.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:47.177 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:47.177 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:47.193 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-16T02:37:45.518980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002855"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640364,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:47.194 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:47.194 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 18, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:37.507435+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:33:37.507435+0000', 'last_peered': '2024-09-16T02:33:37.507435+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:33:37.507435+0000', 'last_undegraded': '2024-09-16T02:33:37.507435+0000', 'last_fullsized': '2024-09-16T02:33:37.507435+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:47.194 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:47.194 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:47.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:47.541 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:47.542 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":527,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-16T02:36:13.496792+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:47.559 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:47.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:47.844 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:47.844 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:47.861 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-16T02:37:47.519463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002931"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":18,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:37.507435+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:33:37.507435+0000","last_peered":"2024-09-16T02:33:37.507435+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:33:37.507435+0000","last_undegraded":"2024-09-16T02:33:37.507435+0000","last_fullsized":"2024-09-16T02:33:37.507435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640364,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:50.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:51.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:51.169 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:51.186 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-16T02:37:49.519924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002959"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640366,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640365,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:51.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:51.469 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:51.485 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-16T02:37:49.519924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002959"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":13,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:33:40.511353+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:33:40.511353+0000","last_peered":"2024-09-16T02:33:40.511353+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:33:40.511353+0000","last_undegraded":"2024-09-16T02:33:40.511353+0000","last_fullsized":"2024-09-16T02:33:40.511353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640366,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640365,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:51.486 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:51.486 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:51.487 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:51.487 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 13, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:33:40.511353+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:33:40.511353+0000', 'last_peered': '2024-09-16T02:33:40.511353+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:33:40.511353+0000', 'last_undegraded': '2024-09-16T02:33:40.511353+0000', 'last_fullsized': '2024-09-16T02:33:40.511353+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:51.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:51.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:51.838 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":531,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:51.854 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:51.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:52.126 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:52.126 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:52.142 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-16T02:37:51.520413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003057"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640366,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640365,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:55.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:55.455 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:55.456 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:55.474 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-16T02:37:53.520772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002792"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640367,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640366,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:55.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:55.783 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:55.784 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:55.801 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-16T02:37:55.521224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640367,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640366,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:55.802 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:37:55.802 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:37:55.802 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:37:55.802 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:37:55.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:37:56.160 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:56.160 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":535,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:37:56.178 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:37:56.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:56.450 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:56.450 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:56.467 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-16T02:37:55.521224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002733"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640367,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640366,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:59.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:37:59.784 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:37:59.784 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:37:59.801 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-16T02:37:59.522104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640368,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640367,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:37:59.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:00.119 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:00.120 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:00.136 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-16T02:37:59.522104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640368,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640367,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:00.137 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:00.137 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:00.137 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:00.137 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:00.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:00.474 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:00.474 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":540,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:00.491 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:00.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:00.769 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-16T02:37:59.522104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":35,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:40.511419+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:33:40.511419+0000","last_peered":"2024-09-16T02:33:40.511419+0000","last_clean":"2024-09-16T02:33:40.511419+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:33:40.511419+0000","last_undegraded":"2024-09-16T02:33:40.511419+0000","last_fullsized":"2024-09-16T02:33:40.511419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640368,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640367,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:03.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:04.098 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:04.098 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:04.115 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-16T02:38:03.522997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002583"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640368,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:04.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:04.395 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:04.396 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:04.412 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-16T02:38:03.522997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002583"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640368,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:04.413 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:04.413 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:04.413 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:04.413 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:04.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:04.767 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:04.767 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":544,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:04.784 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:04.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:05.060 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:05.060 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:05.077 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-16T02:38:03.522997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002583"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640368,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:08.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:08.386 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:08.386 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:08.404 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-16T02:38:07.523913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640368,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:08.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:08.703 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:08.703 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:08.721 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-16T02:38:07.523913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640368,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:08.722 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:08.722 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:08.723 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:08.723 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:08.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:09.081 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:09.082 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":548,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:09.099 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:09.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:09.370 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:09.386 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-16T02:38:07.523913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:33:50.513245+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:33:50.513245+0000","last_peered":"2024-09-16T02:33:50.513245+0000","last_clean":"2024-09-16T02:33:50.513245+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:33:50.513245+0000","last_undegraded":"2024-09-16T02:33:50.513245+0000","last_fullsized":"2024-09-16T02:33:50.513245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640368,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:12.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:12.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:12.694 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:12.712 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-16T02:38:11.524891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002786"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640370,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640369,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:12.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:13.014 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:13.014 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:13.031 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-16T02:38:11.524891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002786"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640370,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640369,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:13.032 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:13.032 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:13.032 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:13.033 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:13.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:13.397 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:13.397 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":552,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:13.415 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:13.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:13.702 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:13.702 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:13.717 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-16T02:38:13.525346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002854"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640371,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640370,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:16.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:17.035 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:17.035 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:17.051 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-16T02:38:15.525661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640371,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640370,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:17.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:17.349 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:17.350 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:17.368 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-16T02:38:15.525661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640371,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640370,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:17.369 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:17.369 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:17.369 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:17.369 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:17.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:17.710 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":557,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:17.727 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:17.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:18.028 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:18.032 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:18.040 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-16T02:38:17.526040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002565"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640371,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640370,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:21.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:21.354 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:21.354 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:21.372 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-16T02:38:19.526360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002446"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640372,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640371,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:21.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:21.678 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:21.696 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-16T02:38:21.526798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002392"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640372,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640371,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:21.697 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:21.697 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:21.697 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:21.697 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:21.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:22.059 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:22.059 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":561,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:22.077 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:22.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:22.367 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:22.368 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:22.384 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-16T02:38:21.526798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002392"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640372,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640371,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:25.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:25.695 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:25.696 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:25.712 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-16T02:38:25.527535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640373,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640372,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:25.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:26.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:26.011 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:26.028 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-16T02:38:25.527535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640373,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640372,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:26.029 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:26.029 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:26.029 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:26.029 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:26.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:26.398 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:26.398 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":565,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:26.416 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:26.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:26.696 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:26.712 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-16T02:38:25.527535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640373,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640372,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:29.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:30.025 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:30.026 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:30.043 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T02:38:29.528301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002260"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640374,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640373,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:30.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:30.342 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:30.343 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:30.360 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T02:38:29.528301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002260"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640374,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640373,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:30.365 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:30.365 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:30.365 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:30.366 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:30.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:30.724 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:30.724 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":570,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:30.742 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:30.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:31.036 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:31.036 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:31.054 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T02:38:29.528301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002260"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640374,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640373,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:34.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:34.367 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:34.367 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:34.383 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-16T02:38:33.529320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002521"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640375,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640374,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:34.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:34.669 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:34.670 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:34.686 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-16T02:38:33.529320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002521"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640375,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640374,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:34.687 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:34.687 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:34.687 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:34.687 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:34.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:35.039 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:35.039 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":574,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:35.056 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:35.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:35.345 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:35.346 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:35.362 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-16T02:38:33.529320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002521"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640375,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640374,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:38.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:38.671 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:38.671 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:38.688 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-16T02:38:37.530278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002742"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640375,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640374,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:38.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:38.982 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:38.998 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-16T02:38:37.530278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002742"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640375,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640374,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:38.998 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:38.999 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:38.999 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:38.999 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:38.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:39.354 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:39.355 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":578,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:39.371 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:39.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:39.651 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:39.651 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:39.668 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-09-16T02:38:39.530738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002945"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640376,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640375,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:42.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:42.975 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:42.990 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-16T02:38:41.531086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640376,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640375,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:42.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:43.264 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:43.280 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-16T02:38:41.531086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640376,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640375,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:43.281 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:43.281 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:43.281 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:43.281 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:43.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:43.646 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:43.646 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":583,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:43.669 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:43.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:43.947 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:43.948 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:43.964 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-16T02:38:43.531527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002709"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640376,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:46.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:47.275 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:47.275 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:47.293 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-16T02:38:45.531882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002562"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640376,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:47.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:47.589 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:47.590 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:47.605 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-16T02:38:47.532306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002505"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640376,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:47.606 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:47.606 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:47.606 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:47.606 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:47.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:47.963 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:47.963 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":587,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:47.980 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:47.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:48.270 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:48.270 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:48.286 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-16T02:38:47.532306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002505"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640376,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:51.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:51.601 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:51.618 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-16T02:38:51.533350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640377,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:51.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:51.907 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:51.908 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:51.923 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-16T02:38:51.533350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640377,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:51.924 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:51.924 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:51.924 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:51.925 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:51.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:52.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:52.255 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":591,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:52.272 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:52.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:52.567 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:52.567 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:52.585 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-16T02:38:51.533350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640377,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:55.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:55.904 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:55.904 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:55.919 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-16T02:38:55.534258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002731"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640379,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640378,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673081,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:55.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:56.215 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:56.215 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:56.232 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-16T02:38:55.534258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002731"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640379,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640378,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673081,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:56.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:38:56.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:38:56.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:38:56.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:38:56.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:38:56.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:56.577 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":596,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:38:56.593 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:38:56.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:38:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:38:56.891 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:38:56.908 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-16T02:38:55.534258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002731"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640379,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640378,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673081,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:38:59.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:00.220 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:00.220 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:00.236 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-16T02:38:59.535258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673082,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:00.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:00.519 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:00.519 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:00.536 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-16T02:38:59.535258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673082,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:00.537 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:00.537 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:00.537 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:00.537 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:00.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:00.886 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:00.886 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":600,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:00.903 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:00.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:01.187 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:01.188 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:01.204 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-16T02:38:59.535258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673082,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:04.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:04.511 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:04.511 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:04.528 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-16T02:39:03.536283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002933"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640380,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673083,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:04.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:04.829 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:04.845 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-16T02:39:03.536283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002933"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640380,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673083,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:04.846 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:04.846 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:04.846 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:04.846 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:04.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:05.188 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:05.188 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":604,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:05.205 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:05.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:05.496 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:05.497 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:05.510 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-16T02:39:03.536283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002933"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640380,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673083,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:08.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:08.821 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:08.821 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:08.839 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-16T02:39:07.537243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002984"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640380,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:08.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:09.137 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:09.138 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:09.155 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-16T02:39:07.537243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002984"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640380,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:09.156 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:09.156 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:09.156 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:09.156 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:09.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:09.507 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:09.507 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":609,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:09.525 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:09.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:09.832 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:09.833 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:09.850 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-16T02:39:09.537684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640381,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:12.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:13.156 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:13.157 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:13.173 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-16T02:39:11.538063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640381,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:13.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:13.449 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:13.449 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:13.465 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-16T02:39:11.538063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640381,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:13.466 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:13.466 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:13.466 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:13.466 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:13.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:13.799 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:13.799 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":613,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:13.816 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:13.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:14.108 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:14.108 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:14.126 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-16T02:39:13.538472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002690"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640382,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673085,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:17.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:17.435 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:17.435 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:17.452 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-16T02:39:15.538897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640382,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673085,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:17.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:17.729 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:17.745 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-16T02:39:17.539432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002637"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640382,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:17.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:17.746 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:17.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:17.746 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:17.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:18.103 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:18.103 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":617,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:18.120 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:18.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:18.407 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:18.407 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:18.422 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-16T02:39:17.539432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002637"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640382,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:21.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:21.732 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:21.748 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-16T02:39:21.540364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002680"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640383,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:21.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:22.032 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:22.033 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:22.049 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-16T02:39:21.540364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002680"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640383,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:22.050 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:22.050 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:22.050 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:22.050 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:22.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:22.392 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:22.392 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":621,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:22.410 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:22.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:22.702 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:22.702 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:22.718 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-16T02:39:21.540364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002680"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640383,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:25.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:26.031 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:26.032 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:26.049 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-16T02:39:25.541295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640384,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673087,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:26.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:26.350 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:26.368 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-16T02:39:25.541295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640384,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673087,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:26.369 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:26.369 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:26.369 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:26.369 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:26.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:26.713 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":626,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:26.729 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:26.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:27.000 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:27.000 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:27.016 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-16T02:39:25.541295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640384,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673087,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:30.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:30.322 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:30.322 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:30.338 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-16T02:39:29.542287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002854"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640385,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:30.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:30.619 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:30.619 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:30.636 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-16T02:39:29.542287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002854"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640385,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:30.637 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:30.637 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:30.637 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:30.637 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:30.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:30.987 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":630,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:31.003 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:31.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:31.273 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:31.273 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:31.289 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-16T02:39:29.542287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002854"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521552+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:35:32.521552+0000","last_peered":"2024-09-16T02:35:32.521552+0000","last_clean":"2024-09-16T02:35:32.521552+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:35:32.521552+0000","last_undegraded":"2024-09-16T02:35:32.521552+0000","last_fullsized":"2024-09-16T02:35:32.521552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640385,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:34.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:34.598 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:34.598 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:34.614 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-16T02:39:33.543301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002937"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640386,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673089,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:34.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:34.898 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:34.914 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-16T02:39:33.543301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002937"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640386,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673089,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:34.914 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:34.914 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:34.915 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:34.915 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:34.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:35.259 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:35.259 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":634,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:35.277 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:35.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:35.549 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:35.549 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:35.565 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-16T02:39:35.543781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640386,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673089,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:38.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:38.872 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:38.872 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:38.889 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-16T02:39:37.544216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640386,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:38.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:39.161 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:39.161 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:39.178 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-16T02:39:37.544216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640386,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:39.179 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:39.179 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:39.179 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:39.179 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:39.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:39.513 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:39.513 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":639,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:39.530 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:39.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:39.826 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:39.842 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-16T02:39:39.544644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002881"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640387,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:42.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:43.151 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:43.151 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:43.170 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-16T02:39:41.544984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002697"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640387,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:43.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:43.470 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:43.470 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:43.487 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-16T02:39:41.544984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002697"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:32.521583+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:35:32.521583+0000","last_peered":"2024-09-16T02:35:32.521583+0000","last_clean":"2024-09-16T02:35:32.521583+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:35:32.521583+0000","last_undegraded":"2024-09-16T02:35:32.521583+0000","last_fullsized":"2024-09-16T02:35:32.521583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640387,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:43.488 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:43.488 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:43.489 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:43.489 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:43.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:43.843 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:43.843 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":643,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:43.860 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:43.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:44.142 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:44.142 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:44.158 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-16T02:39:43.545480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002696"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673091,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:47.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:47.466 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:47.466 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:47.484 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-16T02:39:45.545988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002686"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531380+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:35:35.531380+0000","last_peered":"2024-09-16T02:35:35.531380+0000","last_clean":"2024-09-16T02:35:35.531380+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:35:35.531380+0000","last_undegraded":"2024-09-16T02:35:35.531380+0000","last_fullsized":"2024-09-16T02:35:35.531380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:35.531337+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:35:35.531337+0000","last_peered":"2024-09-16T02:35:35.531337+0000","last_clean":"2024-09-16T02:35:35.531337+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:35:35.531337+0000","last_undegraded":"2024-09-16T02:35:35.531337+0000","last_fullsized":"2024-09-16T02:35:35.531337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673091,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:47.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:47.787 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:47.787 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:47.806 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-16T02:39:47.546520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:47.807 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:47.807 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:47.807 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:47.807 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:47.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:48.167 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":647,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:48.185 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:48.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:48.481 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:48.482 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:48.499 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-16T02:39:47.546520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640388,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:51.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:51.810 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:51.810 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:51.827 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-16T02:39:51.547506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002861"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640389,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:51.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:52.124 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:52.125 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:52.142 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-16T02:39:51.547506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002861"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640389,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:52.142 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:52.143 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 19, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:42.522922+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:35:42.522922+0000', 'last_peered': '2024-09-16T02:35:42.522922+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:35:42.522922+0000', 'last_undegraded': '2024-09-16T02:35:42.522922+0000', 'last_fullsized': '2024-09-16T02:35:42.522922+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:52.143 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:52.143 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:52.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:52.478 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:52.478 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":652,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:52.495 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:52.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:52.787 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:52.788 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:52.804 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-16T02:39:51.547506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002861"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":19,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:42.522922+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:35:42.522922+0000","last_peered":"2024-09-16T02:35:42.522922+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:35:42.522922+0000","last_undegraded":"2024-09-16T02:35:42.522922+0000","last_fullsized":"2024-09-16T02:35:42.522922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640389,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:55.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:56.115 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:56.116 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:56.133 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T02:39:55.548360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640390,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:56.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:56.433 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:56.433 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:56.451 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T02:39:55.548360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640390,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:39:56.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:39:56.452 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:39:56.453 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:39:56.453 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 14, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:35:45.533267+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:35:45.533267+0000', 'last_peered': '2024-09-16T02:35:45.533267+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:35:45.533267+0000', 'last_undegraded': '2024-09-16T02:35:45.533267+0000', 'last_fullsized': '2024-09-16T02:35:45.533267+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:39:56.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:39:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:56.815 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":656,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:39:56.833 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:39:56.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:39:57.120 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:39:57.120 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:39:57.136 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T02:39:55.548360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":14,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:35:45.533267+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:35:45.533267+0000","last_peered":"2024-09-16T02:35:45.533267+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:35:45.533267+0000","last_undegraded":"2024-09-16T02:35:45.533267+0000","last_fullsized":"2024-09-16T02:35:45.533267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640390,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:00.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:00.443 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:00.443 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:00.461 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-16T02:39:59.549287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640391,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:00.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:00.753 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:00.768 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-16T02:39:59.549287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640391,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:00.769 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:00.769 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:00.770 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:00.770 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:00.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:01.131 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:01.131 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":660,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:01.150 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:01.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:01.423 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:01.424 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:01.441 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-16T02:39:59.549287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002766"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640391,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:04.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:04.755 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:04.755 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:04.773 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-16T02:40:03.550309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640392,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:04.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:05.070 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:05.070 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:05.088 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-16T02:40:03.550309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640392,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:05.088 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:05.088 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:05.089 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:05.089 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:05.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:05.423 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:05.423 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":665,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:05.440 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:05.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:05.742 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:05.760 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-16T02:40:05.550830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002975"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640392,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:08.762 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:09.071 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:09.071 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:09.089 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T02:40:07.551272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002911"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640393,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673096,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:09.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:09.381 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:09.398 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T02:40:07.551272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002911"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640393,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673096,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:09.398 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:09.398 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:09.399 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:09.399 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:09.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:09.743 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:09.743 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":669,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:09.760 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:09.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:10.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:10.047 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:10.062 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-16T02:40:09.551694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002916"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":36,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:35:50.534471+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:35:50.534471+0000","last_peered":"2024-09-16T02:35:50.534471+0000","last_clean":"2024-09-16T02:35:50.534471+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:35:50.534471+0000","last_undegraded":"2024-09-16T02:35:50.534471+0000","last_fullsized":"2024-09-16T02:35:50.534471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640394,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640393,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673096,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:13.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:13.370 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:13.370 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:13.386 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T02:40:11.552010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002722"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640394,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640393,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673097,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:13.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:13.686 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:13.687 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:13.705 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-16T02:40:13.552455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002719"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640394,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673097,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:13.705 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:13.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:13.706 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:13.706 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:13.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:14.060 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:14.060 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":673,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:14.076 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:14.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:14.349 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:14.350 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:14.365 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-16T02:40:13.552455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002719"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640394,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673097,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:17.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:17.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:17.672 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:17.688 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-16T02:40:17.553206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002375"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673098,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:17.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:17.965 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:17.966 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:17.981 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-16T02:40:17.553206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002375"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673098,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:17.982 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:17.982 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:17.982 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:17.982 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:17.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:18.338 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:18.338 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":677,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:18.354 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:18.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:18.636 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:18.636 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:18.652 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-16T02:40:17.553206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002375"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:36:00.536418+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:36:00.536418+0000","last_peered":"2024-09-16T02:36:00.536418+0000","last_clean":"2024-09-16T02:36:00.536418+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:36:00.536418+0000","last_undegraded":"2024-09-16T02:36:00.536418+0000","last_fullsized":"2024-09-16T02:36:00.536418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673098,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:21.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:21.960 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:21.960 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:21.977 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-16T02:40:21.554084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002390"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640396,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:21.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:22.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:22.253 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:22.269 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-16T02:40:21.554084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002390"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640396,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:22.269 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:22.269 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:22.270 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:22.270 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:22.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:22.606 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":682,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:22.624 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:22.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:22.916 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:22.916 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:22.933 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-16T02:40:21.554084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002390"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640396,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:25.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:26.246 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:26.246 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:26.262 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-16T02:40:25.555127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640397,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:26.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:26.541 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:26.541 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:26.558 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-16T02:40:25.555127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640397,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:26.559 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:26.559 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:26.559 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:26.559 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:26.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:26.919 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:26.919 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":686,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:26.937 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:26.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:27.219 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:27.220 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:27.237 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-16T02:40:25.555127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640397,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640396,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:30.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:30.550 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:30.551 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:30.569 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-16T02:40:29.556092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640397,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:30.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:30.865 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:30.883 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-16T02:40:29.556092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640397,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:30.883 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:30.884 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:30.884 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:30.884 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:30.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:31.239 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:31.239 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":690,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:31.256 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:31.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:31.554 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:31.554 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:31.572 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-16T02:40:29.556092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640397,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:34.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:34.880 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:34.897 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-16T02:40:33.557201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003116"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640398,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673101,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:34.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:35.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:35.189 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:35.207 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-16T02:40:33.557201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003116"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640398,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673101,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:35.208 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:35.208 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:35.208 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:35.208 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:35.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:35.561 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:35.561 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":695,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:35.577 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:35.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:35.848 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:35.864 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-16T02:40:35.557698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640398,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673101,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:38.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:39.172 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:39.172 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:39.188 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-16T02:40:37.558141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003013"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640399,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:39.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:39.476 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:39.492 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-16T02:40:37.558141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003013"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640399,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:39.493 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:39.493 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:39.493 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:39.493 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:39.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:39.829 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":699,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:39.846 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:39.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:40.118 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:40.119 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:40.135 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-16T02:40:39.558633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003082"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640399,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:43.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:43.449 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:43.449 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:43.465 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-16T02:40:41.559052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002959"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640399,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673103,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:43.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:43.760 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:43.760 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:43.778 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-16T02:40:43.559654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640400,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673103,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:43.778 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:43.778 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:43.779 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:43.779 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:43.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:44.130 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:44.131 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":703,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:44.148 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:44.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:44.423 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:44.424 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:44.440 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-16T02:40:43.559654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640400,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673103,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:47.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:47.750 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:47.750 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:47.767 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-16T02:40:47.560624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640401,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673104,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:47.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:48.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:48.047 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:48.062 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-16T02:40:47.560624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640401,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673104,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:48.063 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:48.063 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:48.064 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:48.064 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:48.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:48.432 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:48.432 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":708,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:48.449 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:48.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:48.756 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:48.757 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:48.774 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-16T02:40:47.560624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640401,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673104,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:51.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:52.087 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:52.087 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:52.104 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-16T02:40:51.561662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003028"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640401,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:52.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:52.396 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:52.396 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:52.412 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-16T02:40:51.561662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003028"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640401,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:52.413 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:52.413 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:52.413 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:52.413 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:52.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:52.777 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:52.777 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":712,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:52.795 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:52.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:53.072 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:53.073 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:53.088 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-16T02:40:51.561662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003028"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640401,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:56.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:56.400 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:56.401 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:56.418 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-16T02:40:55.562815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003161"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640402,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:56.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:56.718 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:56.718 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:56.737 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-16T02:40:55.562815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003161"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640402,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:40:56.738 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:40:56.738 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:40:56.738 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:40:56.738 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:40:56.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:40:57.095 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:57.095 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":716,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:40:57.112 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:40:57.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:40:57.403 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:40:57.403 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:40:57.420 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-16T02:40:55.562815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003161"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640402,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:00.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:00.731 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:00.749 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-16T02:40:59.563825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640404,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673106,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:00.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:01.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:01.051 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:01.067 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-16T02:40:59.563825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640404,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673106,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:01.068 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:01.068 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:01.068 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:01.068 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:01.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:01.419 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:01.419 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":721,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:01.436 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:01.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:01.721 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:01.721 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:01.737 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T02:41:01.564298+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003152"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640404,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673107,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:04.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:05.047 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:05.047 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:05.064 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-16T02:41:03.564647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002984"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640404,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673107,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:05.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:05.337 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:05.338 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:05.353 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-16T02:41:03.564647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002984"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640404,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673107,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:05.354 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:05.354 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:05.355 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:05.355 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:05.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:05.703 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":725,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:05.719 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:05.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:05.999 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:06.000 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:06.016 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-16T02:41:05.565134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002945"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640404,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673107,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:09.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:09.329 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:09.329 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:09.347 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-16T02:41:07.565592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002776"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640405,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673108,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:09.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:09.651 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:09.668 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-16T02:41:09.566055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640405,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673108,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:09.669 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:09.669 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:09.669 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:09.669 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:09.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:10.016 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:10.017 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":729,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:10.033 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:10.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:10.308 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:10.308 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:10.324 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-16T02:41:09.566055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640405,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673108,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:13.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:13.635 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:13.635 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:13.653 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-16T02:41:13.566910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673109,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:13.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:13.932 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:13.948 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-16T02:41:13.566910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673109,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:13.949 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:13.949 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:13.949 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:13.950 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:13.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:14.308 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:14.308 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":733,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:14.325 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:14.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:14.622 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:14.622 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:14.641 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-16T02:41:13.566910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002611"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673109,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:17.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:17.960 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:17.960 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:17.978 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-16T02:41:17.567826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002692"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673110,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:17.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:18.282 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:18.282 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:18.300 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-16T02:41:17.567826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002692"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673110,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:18.301 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:18.301 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:18.301 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:18.301 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:18.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:18.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:18.654 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":738,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:18.672 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:18.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:18.968 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:18.984 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-16T02:41:17.567826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002692"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673110,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:21.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:22.302 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:22.302 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:22.320 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-16T02:41:21.568829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:22.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:22.617 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:22.632 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-16T02:41:21.568829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:22.633 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:22.633 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:22.633 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:22.633 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:22.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:22.966 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:22.966 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":742,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:22.983 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:22.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:23.279 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:23.280 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:23.296 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-16T02:41:21.568829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002774"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:26.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:26.602 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:26.619 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-16T02:41:25.569864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:26.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:26.900 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:26.900 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:26.916 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-16T02:41:25.569864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:26.917 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:26.917 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:26.917 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:26.917 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:26.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:27.275 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:27.275 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":746,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:27.292 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:27.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:27.588 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:27.589 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:27.606 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-16T02:41:27.570336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673112,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:30.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:30.919 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:30.919 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:30.938 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-16T02:41:29.570874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673112,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:30.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:31.234 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:31.234 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:31.250 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-16T02:41:29.570874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673112,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:31.251 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:31.251 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:31.251 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:31.251 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:31.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:31.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:31.606 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":751,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:31.623 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:31.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:31.919 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:31.920 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:31.936 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-16T02:41:31.571352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003038"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:32.539653+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:37:32.539653+0000","last_peered":"2024-09-16T02:37:32.539653+0000","last_clean":"2024-09-16T02:37:32.539653+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:37:32.539653+0000","last_undegraded":"2024-09-16T02:37:32.539653+0000","last_fullsized":"2024-09-16T02:37:32.539653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673113,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:34.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:35.243 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:35.244 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:35.260 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-16T02:41:33.571810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002981"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640410,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673113,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:35.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:35.532 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:35.532 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:35.548 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-16T02:41:33.571810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002981"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640410,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673113,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:35.549 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:35.549 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:35.549 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:35.549 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:35.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:35.904 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:35.965 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":755,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:35.966 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:35.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:36.272 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:36.272 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:36.290 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-16T02:41:35.572263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002907"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640410,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673113,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:39.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:39.603 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:39.604 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:39.621 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T02:41:39.573039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002703"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673114,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:39.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:39.919 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T02:41:39.573039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002703"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673114,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:39.937 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:39.938 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:39.938 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:39.938 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:39.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:40.287 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":759,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:40.303 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:40.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:40.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:40.579 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:40.595 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T02:41:39.573039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002703"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673114,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:43.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:43.908 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:43.908 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:43.925 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-16T02:41:43.573887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002535"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673115,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:43.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:44.220 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:44.237 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-16T02:41:43.573887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002535"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673115,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:44.238 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:44.238 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:44.238 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:44.238 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:44.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:44.604 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:44.605 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":764,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:44.623 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:44.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:44.926 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:44.926 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:44.944 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-16T02:41:43.573887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002535"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:37.540354+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:37:37.540354+0000","last_peered":"2024-09-16T02:37:37.540354+0000","last_clean":"2024-09-16T02:37:37.540354+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:37:37.540354+0000","last_undegraded":"2024-09-16T02:37:37.540354+0000","last_fullsized":"2024-09-16T02:37:37.540354+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673115,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:47.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:48.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:48.255 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:48.273 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-16T02:41:47.574893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:48.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:48.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:48.580 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:48.597 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-16T02:41:47.574893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:48.598 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:48.598 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:48.598 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:48.598 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:48.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:48.945 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:48.945 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":768,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:48.961 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:48.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:49.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:49.244 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:49.261 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-16T02:41:47.574893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002630"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554214+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:37:40.554214+0000","last_peered":"2024-09-16T02:37:40.554214+0000","last_clean":"2024-09-16T02:37:40.554214+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:37:40.554214+0000","last_undegraded":"2024-09-16T02:37:40.554214+0000","last_fullsized":"2024-09-16T02:37:40.554214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:37:40.554166+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:37:40.554166+0000","last_peered":"2024-09-16T02:37:40.554166+0000","last_clean":"2024-09-16T02:37:40.554166+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:37:40.554166+0000","last_undegraded":"2024-09-16T02:37:40.554166+0000","last_fullsized":"2024-09-16T02:37:40.554166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:52.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:52.567 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:52.567 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:52.584 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-16T02:41:51.575919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002879"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673117,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:52.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:52.874 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:52.875 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:52.892 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-16T02:41:51.575919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002879"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673117,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:52.893 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:52.893 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:52.893 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:52.893 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:52.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:53.228 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:53.229 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":772,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:53.246 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:53.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:53.534 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:53.535 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:53.550 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-16T02:41:51.575919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002879"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673117,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:56.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:56.860 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:56.878 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-16T02:41:55.576796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002909"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673117,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:56.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:57.177 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:57.177 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:57.195 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-16T02:41:55.576796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002909"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":20,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:47.541719+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:37:47.541719+0000","last_peered":"2024-09-16T02:37:47.541719+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:37:47.541719+0000","last_undegraded":"2024-09-16T02:37:47.541719+0000","last_fullsized":"2024-09-16T02:37:47.541719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673117,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:41:57.196 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:41:57.196 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 20, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:47.541719+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:37:47.541719+0000', 'last_peered': '2024-09-16T02:37:47.541719+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:37:47.541719+0000', 'last_undegraded': '2024-09-16T02:37:47.541719+0000', 'last_fullsized': '2024-09-16T02:37:47.541719+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:41:57.196 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:41:57.196 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:41:57.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:41:57.562 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:57.562 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":777,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:41:57.581 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:41:57.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:41:57.859 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:41:57.859 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:41:57.876 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-16T02:41:57.577205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002790"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640415,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673118,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:00.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:01.184 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:01.184 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:01.200 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-16T02:41:59.577630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002737"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640415,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673118,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:01.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:01.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:01.497 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:01.514 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-16T02:41:59.577630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002737"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":15,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:37:50.556003+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:37:50.556003+0000","last_peered":"2024-09-16T02:37:50.556003+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:37:50.556003+0000","last_undegraded":"2024-09-16T02:37:50.556003+0000","last_fullsized":"2024-09-16T02:37:50.556003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640415,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673118,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:01.515 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:01.515 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:01.516 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:01.516 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 15, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:37:50.556003+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:37:50.556003+0000', 'last_peered': '2024-09-16T02:37:50.556003+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:37:50.556003+0000', 'last_undegraded': '2024-09-16T02:37:50.556003+0000', 'last_fullsized': '2024-09-16T02:37:50.556003+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:01.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:01.882 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:01.882 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":781,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:01.900 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:01.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:02.181 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:02.182 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:02.198 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-16T02:42:01.578025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640415,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673119,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:05.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:05.511 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:05.511 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:05.529 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-16T02:42:03.578371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640417,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673119,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:05.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:05.825 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:05.826 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:05.842 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-16T02:42:05.578829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002372"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640417,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673119,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:05.843 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:05.843 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:05.843 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:05.843 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:05.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:06.171 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:06.171 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:06.188 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:06.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:06.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:06.469 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:06.485 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-16T02:42:05.578829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002372"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640417,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673119,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:09.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:09.794 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:09.795 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:09.810 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T02:42:09.579655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002449"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673120,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:09.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:10.096 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:10.097 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:10.113 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T02:42:09.579655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002449"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673120,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:10.114 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:10.114 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:10.114 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:10.114 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:10.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:10.475 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:10.475 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:10.494 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:10.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:10.788 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:10.788 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:10.805 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T02:42:09.579655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002449"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673120,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:13.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:14.113 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:14.113 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:14.131 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-16T02:42:13.580528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002503"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640419,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673121,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:14.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:14.429 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:14.430 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:14.446 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-16T02:42:13.580528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002503"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640419,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673121,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:14.447 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:14.447 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:14.447 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:14.447 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:14.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:14.785 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:14.785 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":794,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:14.805 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:14.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:15.078 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:15.078 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:15.094 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-09-16T02:42:13.580528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002503"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640419,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673121,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:18.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:18.407 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:18.407 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:18.425 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-16T02:42:17.581478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002649"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673122,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:18.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:18.725 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:18.726 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:18.746 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-16T02:42:17.581478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002649"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673122,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:18.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:18.746 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:18.747 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:18.747 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:18.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":798,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:19.104 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:19.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:19.398 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:19.398 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:19.415 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-16T02:42:17.581478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002649"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:00.557456+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:38:00.557456+0000","last_peered":"2024-09-16T02:38:00.557456+0000","last_clean":"2024-09-16T02:38:00.557456+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:38:00.557456+0000","last_undegraded":"2024-09-16T02:38:00.557456+0000","last_fullsized":"2024-09-16T02:38:00.557456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673122,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:22.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:22.722 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:22.738 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-16T02:42:21.582506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002851"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673123,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:22.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:23.023 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:23.024 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:23.041 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-16T02:42:21.582506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002851"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673123,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:23.042 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:23.042 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:23.042 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:23.042 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:23.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:23.385 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:23.385 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":802,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:23.402 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:23.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:23.675 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:23.675 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:23.691 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-09-16T02:42:23.582950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640421,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673123,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:26.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:26.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:26.995 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:27.012 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-16T02:42:25.583297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002768"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640421,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673123,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:27.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:27.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:27.304 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:27.320 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-16T02:42:25.583297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002768"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640421,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673123,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:27.320 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:27.320 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:27.321 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:27.321 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:27.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:27.662 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:27.662 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":807,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:27.680 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:27.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:27.964 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:27.964 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:27.980 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-16T02:42:27.583707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002691"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673124,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:30.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:31.290 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:31.290 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:31.308 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-16T02:42:29.584210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002732"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673124,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:31.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:31.582 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:31.582 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:31.599 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-16T02:42:29.584210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002732"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:38:10.559340+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:38:10.559340+0000","last_peered":"2024-09-16T02:38:10.559340+0000","last_clean":"2024-09-16T02:38:10.559340+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:38:10.559340+0000","last_undegraded":"2024-09-16T02:38:10.559340+0000","last_fullsized":"2024-09-16T02:38:10.559340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673124,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:31.600 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:31.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:31.600 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:31.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:31.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:31.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:31.938 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":811,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:31.954 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:31.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:32.251 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:32.268 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-16T02:42:31.585555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003543"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673125,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:35.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:35.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:35.584 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:35.602 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-16T02:42:33.586018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003511"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640423,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673125,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:35.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:35.907 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:35.908 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:35.925 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-16T02:42:35.587053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004103"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640423,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673125,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:35.926 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:35.926 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:35.926 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:35.926 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:35.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:36.292 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:36.292 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":815,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:36.310 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:36.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:36.610 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:36.610 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:36.628 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-09-16T02:42:35.587053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004103"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640423,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673125,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:39.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:39.941 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:39.959 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-16T02:42:39.588002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004295"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640424,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673126,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:39.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:40.248 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:40.249 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:40.265 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-16T02:42:39.588002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004295"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640424,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673126,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:40.266 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:40.266 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:40.266 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:40.266 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:40.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:40.609 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:40.610 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":820,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:40.626 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:40.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:40.909 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:40.909 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:40.924 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-16T02:42:39.588002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004295"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640424,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673126,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:43.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:44.241 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:44.259 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-16T02:42:43.588979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640425,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673127,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:44.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:44.545 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:44.546 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:44.561 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-16T02:42:43.588979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640425,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673127,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:44.563 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:44.563 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:44.563 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:44.563 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:44.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:44.922 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:44.923 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":824,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:44.941 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:44.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:45.241 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:45.258 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-16T02:42:43.588979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640425,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673127,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:48.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:48.568 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:48.569 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:48.585 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-16T02:42:47.589971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673128,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:48.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:48.886 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:48.887 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:48.898 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-16T02:42:47.589971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673128,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:48.899 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:48.900 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:48.900 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:48.900 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:48.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:49.233 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:49.233 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":828,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:49.248 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:49.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:49.518 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:49.518 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:49.534 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-16T02:42:47.589971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673128,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:52.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:52.844 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:52.844 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:52.861 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-16T02:42:51.590900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:52.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:53.143 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:53.144 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:53.159 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-16T02:42:51.590900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:53.160 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:53.160 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:53.160 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:53.160 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:53.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:53.510 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:53.510 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":833,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:53.527 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:53.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:53.812 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:53.812 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:53.828 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-16T02:42:53.591388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002926"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640427,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:56.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:57.135 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:57.136 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:57.152 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-16T02:42:55.591816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002837"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640427,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:57.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:57.426 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:57.427 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:57.443 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-16T02:42:55.591816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002837"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640427,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673129,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:42:57.444 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:42:57.444 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:42:57.444 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:42:57.444 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:42:57.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:42:57.802 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:57.802 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":837,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:42:57.819 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:42:57.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:42:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:42:58.089 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:42:58.105 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-09-16T02:42:57.592217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002703"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640428,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673130,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:01.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:01.419 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:01.419 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:01.436 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-16T02:42:59.592657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002685"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640428,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673130,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:01.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:01.732 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:01.732 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:01.748 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-16T02:43:01.593086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640428,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:01.749 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:01.749 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:01.749 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:01.749 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:01.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:02.111 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:02.112 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":841,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:02.130 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:02.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:02.417 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:02.433 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-16T02:43:01.593086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640428,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:05.434 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:05.748 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:05.766 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-16T02:43:05.593871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002482"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640429,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:05.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:06.061 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:06.062 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:06.078 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-16T02:43:05.593871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002482"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640429,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:06.079 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:06.079 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:06.079 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:06.079 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:06.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:06.421 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":846,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:06.438 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:06.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:06.717 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:06.717 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:06.732 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-16T02:43:05.593871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002482"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640429,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:09.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:10.040 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:10.040 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:10.056 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-16T02:43:09.594748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002530"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673132,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:10.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:10.355 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:10.355 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:10.373 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-16T02:43:09.594748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002530"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673132,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:10.374 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:10.374 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:10.374 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:10.374 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:10.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:10.716 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:10.717 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":850,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:10.733 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:10.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:11.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:11.004 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:11.020 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-16T02:43:09.594748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002530"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673132,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:14.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:14.330 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:14.330 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:14.346 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-16T02:43:13.595713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640431,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640430,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673133,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:14.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:14.617 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:14.618 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:14.634 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-16T02:43:13.595713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640431,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640430,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673133,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:14.634 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:14.635 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:14.635 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:14.635 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:14.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:14.983 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:14.984 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":854,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:15.002 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:15.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:15.288 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:15.289 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:15.306 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-16T02:43:13.595713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002627"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640431,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640430,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673133,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:18.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:18.617 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:18.618 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:18.635 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-16T02:43:17.596815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002943"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673134,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:18.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:18.938 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:18.956 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-16T02:43:17.596815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002943"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673134,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:18.957 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:18.957 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:18.958 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:18.958 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:18.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:19.295 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:19.295 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":858,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:19.312 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:19.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:19.608 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:19.608 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:19.625 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-09-16T02:43:19.597377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003175"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673134,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:22.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:22.935 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:22.936 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:22.953 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-16T02:43:21.597833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:22.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:23.246 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:23.246 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:23.263 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-16T02:43:21.597833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:23.264 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:23.264 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:23.264 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:23.264 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:23.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:23.604 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":863,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:23.622 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:23.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:23.907 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:23.907 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:23.923 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-16T02:43:23.598332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003075"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640433,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640432,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:26.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:27.233 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:27.233 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:27.249 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-16T02:43:25.598821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640433,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640432,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:27.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:27.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:27.528 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-16T02:43:25.598821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640433,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640432,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:27.546 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:27.546 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:27.546 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:27.546 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:27.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:27.890 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:27.891 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":867,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:27.907 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:27.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:28.181 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:28.182 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:28.198 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-09-16T02:43:27.599245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003034"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640434,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673136,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:31.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:31.505 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:31.505 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:31.521 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-16T02:43:29.599563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640434,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673136,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:31.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:31.802 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:31.803 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:31.819 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-16T02:43:31.599996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002619"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640434,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:31.820 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:31.820 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:31.820 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:31.820 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:31.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":871,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:32.175 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:32.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:32.457 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:32.473 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-16T02:43:31.599996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002619"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:32.555524+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:39:32.555524+0000","last_peered":"2024-09-16T02:39:32.555524+0000","last_clean":"2024-09-16T02:39:32.555524+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:39:32.555524+0000","last_undegraded":"2024-09-16T02:39:32.555524+0000","last_fullsized":"2024-09-16T02:39:32.555524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640434,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:35.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:35.785 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:35.802 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-16T02:43:35.601164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640435,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:35.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:36.094 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:36.094 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:36.111 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-16T02:43:35.601164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640435,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:36.112 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:36.112 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:36.112 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:36.112 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:36.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:36.448 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:36.449 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":876,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:36.465 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:36.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:36.763 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:36.763 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:36.781 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-16T02:43:35.601164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640435,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673137,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:39.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:40.082 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:40.082 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:40.097 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-16T02:43:39.602195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640436,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673138,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:40.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:40.389 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:40.405 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-16T02:43:39.602195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640436,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673138,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:40.406 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:40.406 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:40.407 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:40.407 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:40.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:40.745 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:40.745 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":880,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:40.762 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:40.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:41.059 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:41.059 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:41.077 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-16T02:43:39.602195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640436,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673138,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:44.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:44.390 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:44.390 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:44.407 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-16T02:43:43.603197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640437,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673139,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:44.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:44.709 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:44.710 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:44.727 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-16T02:43:43.603197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640437,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673139,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:44.728 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:44.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:44.728 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:44.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:44.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:45.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:45.073 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":884,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:45.093 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:45.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:45.382 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:45.398 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-16T02:43:43.603197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640437,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673139,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:48.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:48.711 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:48.712 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:48.729 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-16T02:43:47.604130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002966"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:48.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:49.024 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:49.025 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:49.042 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-16T02:43:47.604130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002966"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:49.043 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:49.043 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:49.043 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:49.043 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:49.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:49.390 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":888,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:49.408 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:49.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:49.703 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:49.703 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:49.720 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-16T02:43:49.604704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003069"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673140,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:52.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:53.032 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:53.032 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:53.053 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-16T02:43:51.605176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002981"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673141,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:53.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:53.346 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:53.346 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:53.363 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-16T02:43:51.605176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002981"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":32,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:42.557051+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:39:42.557051+0000","last_peered":"2024-09-16T02:39:42.557051+0000","last_clean":"2024-09-16T02:39:42.557051+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:39:42.557051+0000","last_undegraded":"2024-09-16T02:39:42.557051+0000","last_fullsized":"2024-09-16T02:39:42.557051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640438,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673141,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:53.364 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:53.364 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:53.365 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:53.365 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:53.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:53.697 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:53.697 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":893,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:53.714 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:53.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:53.998 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:53.999 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:54.014 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-16T02:43:53.605660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640439,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640438,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673141,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:57.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:57.335 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:57.335 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:57.352 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-16T02:43:55.606145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002948"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574621+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:39:45.574621+0000","last_peered":"2024-09-16T02:39:45.574621+0000","last_clean":"2024-09-16T02:39:45.574621+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:39:45.574621+0000","last_undegraded":"2024-09-16T02:39:45.574621+0000","last_fullsized":"2024-09-16T02:39:45.574621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:39:45.574583+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:39:45.574583+0000","last_peered":"2024-09-16T02:39:45.574583+0000","last_clean":"2024-09-16T02:39:45.574583+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:39:45.574583+0000","last_undegraded":"2024-09-16T02:39:45.574583+0000","last_fullsized":"2024-09-16T02:39:45.574583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640439,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640438,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673141,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:57.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:57.645 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:57.645 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:57.662 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-16T02:43:57.606597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640440,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673142,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:43:57.662 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:43:57.663 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:43:57.663 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:43:57.663 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:43:57.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:43:57.996 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:57.997 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":897,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:43:58.013 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:43:58.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:43:58.287 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:43:58.287 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:43:58.303 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-16T02:43:57.606597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640440,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673142,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:01.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:01.613 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:01.613 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:01.631 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-16T02:44:01.607583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002878"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640440,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:01.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:01.922 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:01.922 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:01.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-16T02:44:01.607583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002878"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640440,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:01.940 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:01.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 21, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:52.558022+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:39:52.558022+0000', 'last_peered': '2024-09-16T02:39:52.558022+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:39:52.558022+0000', 'last_undegraded': '2024-09-16T02:39:52.558022+0000', 'last_fullsized': '2024-09-16T02:39:52.558022+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:01.940 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:01.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:01.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:02.282 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":901,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:02.298 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:02.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:02.592 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:02.593 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:02.610 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-16T02:44:01.607583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002878"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":21,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:52.558022+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:39:52.558022+0000","last_peered":"2024-09-16T02:39:52.558022+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:39:52.558022+0000","last_undegraded":"2024-09-16T02:39:52.558022+0000","last_fullsized":"2024-09-16T02:39:52.558022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640440,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:05.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:05.916 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:05.917 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:05.933 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-16T02:44:05.608609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640441,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640440,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:05.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:06.227 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:06.227 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:06.244 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-16T02:44:05.608609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640441,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640440,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:06.245 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:06.245 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:06.245 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:06.245 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 16, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:39:55.576464+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:39:55.576464+0000', 'last_peered': '2024-09-16T02:39:55.576464+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:39:55.576464+0000', 'last_undegraded': '2024-09-16T02:39:55.576464+0000', 'last_fullsized': '2024-09-16T02:39:55.576464+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:06.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:06.588 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:06.589 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":906,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:06.605 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:06.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:06.886 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:06.886 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:06.903 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-16T02:44:05.608609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:39:55.576464+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:39:55.576464+0000","last_peered":"2024-09-16T02:39:55.576464+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:39:55.576464+0000","last_undegraded":"2024-09-16T02:39:55.576464+0000","last_fullsized":"2024-09-16T02:39:55.576464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640441,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640440,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:09.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:10.215 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:10.215 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:10.233 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-16T02:44:09.609542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002944"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640442,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673144,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:10.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:10.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:10.528 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:10.543 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-16T02:44:09.609542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002944"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640442,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673144,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:10.544 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:10.544 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:10.545 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:10.545 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:10.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:10.890 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:10.890 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":910,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:10.906 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:10.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:11.180 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:11.180 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:11.196 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-16T02:44:09.609542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002944"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640442,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673144,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:14.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:14.504 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:14.520 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T02:44:13.610570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002986"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640443,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673145,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:14.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:14.823 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:14.823 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:14.841 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T02:44:13.610570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002986"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640443,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673145,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:14.842 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:14.842 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:14.842 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:14.842 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:14.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:15.203 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:15.203 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":914,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:15.220 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:15.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:15.518 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:15.518 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:15.535 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T02:44:13.610570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002986"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640443,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673145,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:18.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:18.844 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:18.844 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:18.862 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T02:44:17.611588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640444,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640443,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673146,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:18.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:19.161 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:19.178 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T02:44:17.611588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640444,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640443,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673146,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:19.179 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:19.179 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:19.179 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:19.180 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:19.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:19.512 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:19.512 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":919,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:19.528 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:19.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:19.818 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:19.818 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:19.834 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-16T02:44:19.612160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640444,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640443,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673146,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:22.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:23.142 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:23.142 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:23.159 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-16T02:44:21.612573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003031"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640444,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640443,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:23.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:23.438 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:23.438 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:23.454 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-16T02:44:21.612573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003031"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640444,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640443,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:23.455 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:23.455 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:23.455 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:23.455 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:23.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:23.817 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:23.817 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":923,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:23.836 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:23.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:24.132 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:24.133 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:24.149 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-16T02:44:23.613047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003017"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640445,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:27.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:27.456 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:27.456 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:27.473 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-16T02:44:25.613382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002812"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640445,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:27.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:27.761 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:27.762 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:27.779 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-16T02:44:27.613846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:27.779 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:27.780 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:27.780 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:27.780 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:27.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:28.109 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:28.109 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":927,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:28.126 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:28.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:28.397 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:28.414 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-16T02:44:27.613846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":38,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:10.578605+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:40:10.578605+0000","last_peered":"2024-09-16T02:40:10.578605+0000","last_clean":"2024-09-16T02:40:10.578605+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:40:10.578605+0000","last_undegraded":"2024-09-16T02:40:10.578605+0000","last_fullsized":"2024-09-16T02:40:10.578605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:31.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:31.724 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:31.725 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:31.742 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-16T02:44:31.614738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673149,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:31.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:32.041 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:32.042 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:32.059 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-16T02:44:31.614738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673149,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:32.060 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:32.060 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:32.060 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:32.061 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:32.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:32.409 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":932,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:32.426 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:32.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:32.696 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:32.696 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:32.712 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-16T02:44:31.614738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640446,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673149,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:35.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:36.025 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:36.025 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:36.042 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-16T02:44:35.615765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640447,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673149,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:36.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:36.335 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:36.335 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:36.352 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-16T02:44:35.615765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640447,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673149,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:36.353 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:36.353 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:36.353 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:36.353 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:36.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:36.696 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":936,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:36.711 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:36.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:36.994 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:37.011 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-16T02:44:35.615765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640447,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673149,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:40.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:40.319 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:40.335 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T02:44:39.616759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002913"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640448,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673151,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:40.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:40.628 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:40.628 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:40.645 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T02:44:39.616759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002913"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640448,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673151,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:40.646 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:40.646 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:40.646 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:40.646 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:40.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:40.991 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:40.991 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":940,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:41.008 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:41.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:41.294 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:41.294 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:41.311 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T02:44:39.616759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002913"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:40:20.580515+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:40:20.580515+0000","last_peered":"2024-09-16T02:40:20.580515+0000","last_clean":"2024-09-16T02:40:20.580515+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:40:20.580515+0000","last_undegraded":"2024-09-16T02:40:20.580515+0000","last_fullsized":"2024-09-16T02:40:20.580515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640448,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673151,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:44.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:44.622 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:44.623 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:44.639 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-16T02:44:43.617745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640449,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:44.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:44.940 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:44.941 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:44.959 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-16T02:44:43.617745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640449,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:44.960 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:44.960 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:44.960 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:44.960 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:44.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:45.313 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":944,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:45.330 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:45.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:45.615 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:45.615 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:45.632 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-16T02:44:43.617745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640449,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:48.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:48.943 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:48.944 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:48.962 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-16T02:44:47.619446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003681"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640450,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673153,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:48.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:49.257 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:49.258 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:49.274 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-16T02:44:47.619446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003681"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640450,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673153,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:49.275 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:49.275 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:49.276 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:49.276 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:49.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:49.634 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:49.635 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":949,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:49.652 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:49.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:49.939 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:49.940 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:49.956 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-09-16T02:44:49.619949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003686"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640450,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673153,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:52.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:53.266 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:53.266 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:53.283 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-16T02:44:51.620514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003754"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640450,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:53.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:53.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:53.581 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:53.597 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-16T02:44:51.620514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003754"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640450,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:53.598 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:53.598 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:53.598 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:53.598 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:53.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:53.932 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:53.932 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":953,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:53.949 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:53.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:54.228 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:54.228 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:54.244 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-09-16T02:44:53.621053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640451,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:57.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:57.548 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:57.548 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:57.564 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-16T02:44:55.621498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003752"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640451,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:57.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:57.863 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:57.864 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:57.881 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-16T02:44:57.622008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640452,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673155,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:44:57.882 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:44:57.882 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:44:57.882 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:44:57.882 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:44:57.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:44:58.222 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:58.222 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":957,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:44:58.238 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:44:58.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:44:58.516 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:44:58.517 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:44:58.533 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-16T02:44:57.622008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640452,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673155,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:01.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:01.850 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:01.851 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:01.869 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T02:45:01.623105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003156"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640452,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:01.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:02.162 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:02.163 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:02.179 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T02:45:01.623105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003156"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640452,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:02.180 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:02.180 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:02.180 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:02.180 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:02.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:02.522 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:02.522 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":962,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:45:02.539 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:02.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:02.828 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:02.828 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:02.846 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T02:45:01.623105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003156"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640452,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:05.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:06.159 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:06.159 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:06.177 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-16T02:45:05.624173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003120"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640453,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:06.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:06.471 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:06.472 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:06.488 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-16T02:45:05.624173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003120"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640453,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:06.489 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:06.489 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:06.489 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:06.489 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:06.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:06.849 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:06.850 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":966,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T02:37:49.520003+0000","services":{}},"progress_events":{}} 2024-09-16T02:45:06.867 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:06.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:07.146 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:07.146 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:07.163 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-16T02:45:05.624173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003120"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640453,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673156,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:10.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:10.472 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:10.473 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:10.492 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-16T02:45:09.625181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003173"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640454,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673157,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:10.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:10.798 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:10.798 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:10.815 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-16T02:45:09.625181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003173"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640454,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673157,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:10.816 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:10.816 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:10.816 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:10.816 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:10.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:11.182 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:11.183 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":970,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T02:45:07.624714+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:45:11.201 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:11.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:11.498 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:11.513 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-16T02:45:09.625181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003173"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640454,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673157,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:14.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:14.834 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:14.835 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:14.853 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-16T02:45:13.626180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003074"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640455,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673158,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:14.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:15.158 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:15.159 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:15.177 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-16T02:45:13.626180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003074"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640455,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673158,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:15.177 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:15.177 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:15.178 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:15.178 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:15.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:15.535 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:15.535 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":975,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T02:45:11.625757+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:45:15.551 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:15.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:15.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:15.848 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:15.864 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-09-16T02:45:15.626676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003051"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640455,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673158,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:18.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:19.177 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:19.195 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-16T02:45:17.627058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640456,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673159,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:19.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:19.487 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:19.487 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:19.505 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-16T02:45:17.627058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640456,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673159,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:19.506 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:19.506 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:19.506 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:19.506 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:19.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:19.854 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":979,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T02:45:11.625757+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:45:19.871 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:19.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:20.144 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:20.145 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:20.161 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-16T02:45:19.627493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002852"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640456,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673159,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:23.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:23.479 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:23.480 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:23.497 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-16T02:45:21.627850+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002668"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640456,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673160,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:23.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:23.800 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:23.801 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:23.818 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-16T02:45:23.628378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640457,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673160,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:23.819 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:23.819 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:23.819 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:23.820 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:23.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:24.183 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:24.184 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":983,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T02:45:11.625757+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:45:24.202 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:24.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:24.496 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:24.496 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:24.514 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-16T02:45:23.628378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002689"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640457,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673160,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:27.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:27.823 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:27.823 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:27.839 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-16T02:45:27.629219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002543"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673161,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:27.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:28.130 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:28.130 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:28.147 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-16T02:45:27.629219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002543"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673161,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:28.147 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:28.147 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:28.148 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:28.148 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:28.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:28.510 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":988,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T02:45:11.625757+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:45:28.528 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:28.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:28.811 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:28.811 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:28.827 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-16T02:45:27.629219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002543"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673161,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:31.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:32.140 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:32.157 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-16T02:45:31.630102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002609"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:32.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:32.460 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:32.478 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-16T02:45:31.630102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002609"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:32.479 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:32.479 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:32.479 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:32.479 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:32.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:32.819 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:32.819 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":992,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T02:45:11.625757+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:45:32.836 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:32.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:33.114 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:33.114 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:33.130 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-16T02:45:31.630102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002609"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:32.571216+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:41:32.571216+0000","last_peered":"2024-09-16T02:41:32.571216+0000","last_clean":"2024-09-16T02:41:32.571216+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:41:32.571216+0000","last_undegraded":"2024-09-16T02:41:32.571216+0000","last_fullsized":"2024-09-16T02:41:32.571216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:36.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:36.439 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:36.439 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:36.455 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-16T02:45:35.631090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002712"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:36.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:36.726 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:36.727 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:36.743 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-16T02:45:35.631090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002712"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:36.743 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:36.743 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:36.743 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:36.744 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:36.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:37.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:37.073 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":996,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-16T02:45:11.625757+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:45:37.091 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:37.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:37.387 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:37.388 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:37.406 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-16T02:45:35.631090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002712"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:40.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:40.717 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:40.718 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:40.735 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-16T02:45:39.632070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640460,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673163,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:40.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:41.030 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:41.031 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:41.047 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-16T02:45:39.632070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640460,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673163,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:41.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:41.048 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:41.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:41.048 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:41.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:41.398 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:41.398 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1000,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:45:41.416 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:41.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:41.687 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:41.688 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:41.704 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-16T02:45:41.632534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640460,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673164,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:44.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:45.013 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:45.013 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:45.031 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-16T02:45:43.633016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640461,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673164,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:45.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:45.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:45.324 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:45.342 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-16T02:45:43.633016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640461,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673164,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:45.343 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:45.343 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:45.343 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:45.343 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:45.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:45.693 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:45.693 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1005,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:45:45.709 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:45.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:45.988 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:45.989 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:46.005 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-09-16T02:45:45.633509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640461,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673164,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:49.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:49.310 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:49.326 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-16T02:45:47.633997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640462,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673165,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:49.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:49.611 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:49.611 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:49.629 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-16T02:45:47.633997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002906"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640462,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673165,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:49.630 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:49.630 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:49.630 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:49.630 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:49.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:49.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:49.995 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1009,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:45:50.013 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:50.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:50.304 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:50.305 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:50.322 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-09-16T02:45:49.634433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002865"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640462,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673165,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:53.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:53.634 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:53.635 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:53.652 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-16T02:45:53.635469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640463,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673166,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:53.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:53.928 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:53.929 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:53.945 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-16T02:45:53.635469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640463,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673166,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:53.946 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:53.946 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:53.946 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:53.946 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:53.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:54.308 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:54.308 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1013,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:45:54.326 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:54.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:54.598 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:54.598 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:54.614 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-16T02:45:53.635469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":33,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:47.573353+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:41:47.573353+0000","last_peered":"2024-09-16T02:41:47.573353+0000","last_clean":"2024-09-16T02:41:47.573353+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:41:47.573353+0000","last_undegraded":"2024-09-16T02:41:47.573353+0000","last_fullsized":"2024-09-16T02:41:47.573353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640463,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673166,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:57.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:57.924 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:57.924 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:57.940 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-16T02:45:57.636378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002868"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673167,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:57.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:58.239 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:58.240 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:58.257 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-16T02:45:57.636378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002868"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673167,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:45:58.258 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:45:58.258 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:45:58.258 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:45:58.258 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:45:58.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:45:58.609 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:58.609 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1018,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:45:58.627 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:45:58.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:45:58.929 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:45:58.930 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:45:58.947 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-16T02:45:57.636378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002868"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594739+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:41:50.594739+0000","last_peered":"2024-09-16T02:41:50.594739+0000","last_clean":"2024-09-16T02:41:50.594739+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:41:50.594739+0000","last_undegraded":"2024-09-16T02:41:50.594739+0000","last_fullsized":"2024-09-16T02:41:50.594739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:41:50.594695+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:41:50.594695+0000","last_peered":"2024-09-16T02:41:50.594695+0000","last_clean":"2024-09-16T02:41:50.594695+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:41:50.594695+0000","last_undegraded":"2024-09-16T02:41:50.594695+0000","last_fullsized":"2024-09-16T02:41:50.594695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673167,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:01.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:02.262 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:02.280 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-16T02:46:01.637383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:02.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:02.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:02.581 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:02.599 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-16T02:46:01.637383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:02.600 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:46:02.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:46:02.600 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:46:02.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:46:02.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:46:02.946 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:02.946 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1022,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:46:02.964 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:46:02.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:03.247 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:03.248 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:03.263 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-16T02:46:01.637383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:06.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:06.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:06.578 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:06.596 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-16T02:46:05.638391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640465,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:06.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:06.894 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:06.894 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:06.912 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-16T02:46:05.638391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640465,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:06.912 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:46:06.913 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 22, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:41:57.574404+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:41:57.574404+0000', 'last_peered': '2024-09-16T02:41:57.574404+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:41:57.574404+0000', 'last_undegraded': '2024-09-16T02:41:57.574404+0000', 'last_fullsized': '2024-09-16T02:41:57.574404+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:46:06.913 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:46:06.913 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:46:06.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:46:07.260 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:07.261 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1026,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:46:07.277 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:46:07.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:07.549 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:07.550 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:07.566 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-16T02:46:05.638391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":22,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:41:57.574404+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:41:57.574404+0000","last_peered":"2024-09-16T02:41:57.574404+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:41:57.574404+0000","last_undegraded":"2024-09-16T02:41:57.574404+0000","last_fullsized":"2024-09-16T02:41:57.574404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640465,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:10.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:10.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:10.880 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:10.897 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-16T02:46:09.639421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":23,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:44:02.590962+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:44:02.590962+0000","last_peered":"2024-09-16T02:44:02.590962+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:44:02.590962+0000","last_undegraded":"2024-09-16T02:44:02.590962+0000","last_fullsized":"2024-09-16T02:44:02.590962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640466,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673169,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:10.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:11.190 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:11.190 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:11.207 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-16T02:46:09.639421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54368,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689312,"statfs":{"total":193273528320,"available":193217855488,"internally_reserved":0,"allocated":327680,"data_stored":86310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":23,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:44:02.590962+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:44:02.590962+0000","last_peered":"2024-09-16T02:44:02.590962+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:44:02.590962+0000","last_undegraded":"2024-09-16T02:44:02.590962+0000","last_fullsized":"2024-09-16T02:44:02.590962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":17,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:42:00.596584+0000","last_change":"2024-09-16T02:31:35.490032+0000","last_active":"2024-09-16T02:42:00.596584+0000","last_peered":"2024-09-16T02:42:00.596584+0000","last_clean":"2024-09-16T02:29:25.235394+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:42:00.596584+0000","last_undegraded":"2024-09-16T02:42:00.596584+0000","last_fullsized":"2024-09-16T02:42:00.596584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640466,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673169,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705873,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:11.208 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:46:11.208 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "14'1", 'reported_seq': 23, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:44:02.590962+0000', 'last_change': '2024-09-16T02:31:32.491975+0000', 'last_active': '2024-09-16T02:44:02.590962+0000', 'last_peered': '2024-09-16T02:44:02.590962+0000', 'last_clean': '2024-09-16T02:29:16.448904+0000', 'last_became_active': '2024-09-16T02:29:29.546635+0000', 'last_became_peered': '2024-09-16T02:29:29.546635+0000', 'last_unstale': '2024-09-16T02:44:02.590962+0000', 'last_undegraded': '2024-09-16T02:44:02.590962+0000', 'last_fullsized': '2024-09-16T02:44:02.590962+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:46:11.208 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:46:11.208 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "16'2", 'reported_seq': 17, 'reported_epoch': 19, 'state': 'active+recovering', 'last_fresh': '2024-09-16T02:42:00.596584+0000', 'last_change': '2024-09-16T02:31:35.490032+0000', 'last_active': '2024-09-16T02:42:00.596584+0000', 'last_peered': '2024-09-16T02:42:00.596584+0000', 'last_clean': '2024-09-16T02:29:25.235394+0000', 'last_became_active': '2024-09-16T02:29:29.553972+0000', 'last_became_peered': '2024-09-16T02:29:29.553972+0000', 'last_unstale': '2024-09-16T02:42:00.596584+0000', 'last_undegraded': '2024-09-16T02:42:00.596584+0000', 'last_fullsized': '2024-09-16T02:42:00.596584+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T02:46:11.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:46:11.542 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:11.543 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1031,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726453768,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":55672832,"bytes_avail":193217855488,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:46:11.559 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T02:46:11.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:11.856 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:11.856 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:11.874 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-09-16T02:46:11.639848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54372,"kb_used_data":324,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689308,"statfs":{"total":193273528320,"available":193217851392,"internally_reserved":0,"allocated":331776,"data_stored":86329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":-1,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002958"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":23,"reported_epoch":19,"state":"active+recovering","last_fresh":"2024-09-16T02:44:02.590962+0000","last_change":"2024-09-16T02:31:32.491975+0000","last_active":"2024-09-16T02:44:02.590962+0000","last_peered":"2024-09-16T02:44:02.590962+0000","last_clean":"2024-09-16T02:29:16.448904+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:44:02.590962+0000","last_undegraded":"2024-09-16T02:44:02.590962+0000","last_fullsized":"2024-09-16T02:44:02.590962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:46:08.318207+0000","last_change":"2024-09-16T02:46:08.316839+0000","last_active":"2024-09-16T02:46:08.318207+0000","last_peered":"2024-09-16T02:46:08.318207+0000","last_clean":"2024-09-16T02:46:08.318207+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:46:08.318207+0000","last_undegraded":"2024-09-16T02:46:08.318207+0000","last_fullsized":"2024-09-16T02:46:08.318207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:14:50.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640466,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":43155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673170,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":43174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:14.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:46:15.187 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:46:15.187 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:46:15.205 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-16T02:46:13.640335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":54376,"kb_used_data":328,"kb_used_omap":3,"kb_used_meta":54012,"kb_avail":188689304,"statfs":{"total":193273528320,"available":193217847296,"internally_reserved":0,"allocated":335872,"data_stored":86348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":55309196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":-1,"num_objects_degraded":0,"num_objects_misplaced":-3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002952"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:30.601386+0000","last_change":"2024-09-16T02:29:29.549765+0000","last_active":"2024-09-16T02:42:30.601386+0000","last_peered":"2024-09-16T02:42:30.601386+0000","last_clean":"2024-09-16T02:42:30.601386+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:42:30.601386+0000","last_undegraded":"2024-09-16T02:42:30.601386+0000","last_fullsized":"2024-09-16T02:42:30.601386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:39.892980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:42:20.599567+0000","last_change":"2024-09-16T02:29:29.554089+0000","last_active":"2024-09-16T02:42:20.599567+0000","last_peered":"2024-09-16T02:42:20.599567+0000","last_clean":"2024-09-16T02:42:20.599567+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:42:20.599567+0000","last_undegraded":"2024-09-16T02:42:20.599567+0000","last_fullsized":"2024-09-16T02:42:20.599567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.083686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":34,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:52.589625+0000","last_change":"2024-09-16T02:29:19.776575+0000","last_active":"2024-09-16T02:43:52.589625+0000","last_peered":"2024-09-16T02:43:52.589625+0000","last_clean":"2024-09-16T02:43:52.589625+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:43:52.589625+0000","last_undegraded":"2024-09-16T02:43:52.589625+0000","last_fullsized":"2024-09-16T02:43:52.589625+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:30:47.165094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615737+0000","last_change":"2024-09-16T02:29:29.550050+0000","last_active":"2024-09-16T02:43:55.615737+0000","last_peered":"2024-09-16T02:43:55.615737+0000","last_clean":"2024-09-16T02:43:55.615737+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:43:55.615737+0000","last_undegraded":"2024-09-16T02:43:55.615737+0000","last_fullsized":"2024-09-16T02:43:55.615737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:13.921305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:46:07.864245+0000","last_change":"2024-09-16T02:46:07.864025+0000","last_active":"2024-09-16T02:46:07.864245+0000","last_peered":"2024-09-16T02:46:07.864245+0000","last_clean":"2024-09-16T02:46:07.864245+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:46:07.864245+0000","last_undegraded":"2024-09-16T02:46:07.864245+0000","last_fullsized":"2024-09-16T02:46:07.864245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:27:59.176608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:46:08.318207+0000","last_change":"2024-09-16T02:46:08.316839+0000","last_active":"2024-09-16T02:46:08.318207+0000","last_peered":"2024-09-16T02:46:08.318207+0000","last_clean":"2024-09-16T02:46:08.318207+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:46:08.318207+0000","last_undegraded":"2024-09-16T02:46:08.318207+0000","last_fullsized":"2024-09-16T02:46:08.318207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:14:50.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:55.615656+0000","last_change":"2024-09-16T02:29:29.554030+0000","last_active":"2024-09-16T02:43:55.615656+0000","last_peered":"2024-09-16T02:43:55.615656+0000","last_clean":"2024-09-16T02:43:55.615656+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:43:55.615656+0000","last_undegraded":"2024-09-16T02:43:55.615656+0000","last_fullsized":"2024-09-16T02:43:55.615656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:23.969795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T02:43:32.586749+0000","last_change":"2024-09-16T02:29:29.546290+0000","last_active":"2024-09-16T02:43:32.586749+0000","last_peered":"2024-09-16T02:43:32.586749+0000","last_clean":"2024-09-16T02:43:32.586749+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:43:32.586749+0000","last_undegraded":"2024-09-16T02:43:32.586749+0000","last_fullsized":"2024-09-16T02:43:32.586749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:55:49.596682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":11,"seq":47244640467,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":43174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":47244640466,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":42949673170,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705874,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":43174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:46:15.206 INFO:tasks.osd_recovery.ceph_manager:clean! 2024-09-16T02:46:15.206 INFO:tasks.osd_recovery:rados -p rbd bench 20 write -b 1 --no-cleanup 2024-09-16T02:46:15.206 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd bench 20 write -b 1 --no-cleanup 2024-09-16T02:46:15.244 INFO:teuthology.orchestra.run.smithi137.stdout:hints = 1 2024-09-16T02:46:15.244 INFO:teuthology.orchestra.run.smithi137.stdout:Maintaining 16 concurrent writes of 1 bytes to objects of size 1 for up to 20 seconds or 0 objects 2024-09-16T02:46:15.244 INFO:teuthology.orchestra.run.smithi137.stdout:Object prefix: benchmark_data_smithi137_57209 2024-09-16T02:46:15.244 INFO:teuthology.orchestra.run.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T02:46:15.244 INFO:teuthology.orchestra.run.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-16T02:46:15.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:15.272+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x559d21a2e000 legacy=0x559d21a304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:15.282 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:15.280+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a27650 legacy=0x559d21a29a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:46:16.244 INFO:teuthology.orchestra.run.smithi137.stdout: 1 16 2682 2666 0.00254236 0.0025425 0.00588541 0.00598015 2024-09-16T02:46:16.581 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:16.580+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f490404dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:16.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:16.588+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a27650 legacy=0x7f4904045d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:16.594 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:16.592+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f4904046490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:46:16.737 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:16.736+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a27650 legacy=0x7f4904045d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:17.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2 16 5157 5141 0.00245126 0.00236034 0.00723381 0.00621631 2024-09-16T02:46:17.557 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:17.556+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x559d21a2e000 legacy=0x7f4904046490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:18.244 INFO:teuthology.orchestra.run.smithi137.stdout: 3 16 7369 7353 0.00233729 0.00210953 0.00771972 0.00651831 2024-09-16T02:46:18.333 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:18.332+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f49040420f0 legacy=0x7f49040444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:18.554 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:18.552+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f4904046450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:18.946 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:18.944+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x559d21a27650 legacy=0x7f4904045d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:19.244 INFO:teuthology.orchestra.run.smithi137.stdout: 4 16 9428 9412 0.00224383 0.00196362 0.00693337 0.00679112 2024-09-16T02:46:19.755 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:19.756+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f49040420f0 legacy=0x7f49040444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:46:20.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 16 11385 11369 0.0021683 0.00186634 0.00643731 0.00702684 2024-09-16T02:46:21.244 INFO:teuthology.orchestra.run.smithi137.stdout: 6 15 13128 13113 0.00208409 0.00166321 0.00441475 0.00731423 2024-09-16T02:46:21.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:21.684+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f490404d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:22.245 INFO:teuthology.orchestra.run.smithi137.stdout: 7 16 14875 14859 0.00202397 0.00166512 0.00913035 0.00753017 2024-09-16T02:46:22.935 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:22.932+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f4904045d00 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:46:23.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:23.148+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f4904045d00 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:23.247 INFO:teuthology.orchestra.run.smithi137.stdout: 8 16 16542 16526 0.00196936 0.00158978 0.0083661 0.0077398 2024-09-16T02:46:24.247 INFO:teuthology.orchestra.run.smithi137.stdout: 9 16 18092 18076 0.00191479 0.0014782 0.0142928 0.00795775 2024-09-16T02:46:25.104 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:25.104+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f490404cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:25.155 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:25.156+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f49040420f0 legacy=0x7f49040444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:25.214 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:25.212+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f490404cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:25.247 INFO:teuthology.orchestra.run.smithi137.stdout: 10 16 19575 19559 0.00186474 0.0014143 0.00726353 0.0081728 2024-09-16T02:46:25.742 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:25.740+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f49040420f0 legacy=0x7f49040444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:26.222 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:26.220+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f4904045d00 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:26.247 INFO:teuthology.orchestra.run.smithi137.stdout: 11 16 20960 20944 0.0018153 0.00132084 0.00748369 0.00839613 2024-09-16T02:46:26.281 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:26.280+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f490404cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:26.979 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:26.976+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f490404e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:27.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:27.180+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f4904045d00 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:27.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:27.180+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f49040420f0 legacy=0x7f49040444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:27.247 INFO:teuthology.orchestra.run.smithi137.stdout: 12 16 22332 22316 0.00177306 0.00130844 0.00655394 0.0085967 2024-09-16T02:46:27.295 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:27.296+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x559d21a2e000 legacy=0x7f490404e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:27.329 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:27.328+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f4904045d00 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:27.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:27.892+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f4904045d00 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:46:28.247 INFO:teuthology.orchestra.run.smithi137.stdout: 13 16 23671 23655 0.0017349 0.00127697 0.00662802 0.00878783 2024-09-16T02:46:28.465 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:28.464+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x559d21a2e000 legacy=0x7f490404e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:46:29.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:29.032+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f4904045d00 legacy=0x7f490404a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:29.247 INFO:teuthology.orchestra.run.smithi137.stdout: 14 16 24925 24909 0.0016964 0.00119591 0.0127034 0.00898606 2024-09-16T02:46:29.930 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:29.928+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f4904045d00 legacy=0x7f490404cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:29.983 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:29.980+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f490404e450 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:30.247 INFO:teuthology.orchestra.run.smithi137.stdout: 15 16 26171 26155 0.00166252 0.00118828 0.00826669 0.00916849 2024-09-16T02:46:30.468 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:30.468+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f4904045d00 legacy=0x7f490404cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:31.019 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:31.020+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f49040420f0 legacy=0x7f49040444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:31.248 INFO:teuthology.orchestra.run.smithi137.stdout: 16 16 27399 27383 0.00163173 0.00117111 0.00484836 0.00934442 2024-09-16T02:46:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: 17 16 28557 28541 0.00160063 0.00110435 0.00924201 0.00952544 2024-09-16T02:46:32.480 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:32.480+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f49040420f0 legacy=0x7f49040444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:33.249 INFO:teuthology.orchestra.run.smithi137.stdout: 18 16 29703 29687 0.00157242 0.00109291 0.0167602 0.009695 2024-09-16T02:46:33.251 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:33.252+0000 7f4921a33640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6808/2156154695 conn(0x7f490404e450 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:46:33.333 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:33.332+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f4904045d00 legacy=0x7f490404cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:33.988 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:33.988+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f490404e450 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:34.249 INFO:teuthology.orchestra.run.smithi137.stdout: 19 16 30823 30807 0.00154588 0.00106812 0.0239053 0.00986369 2024-09-16T02:46:34.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-09-16T02:46:34.796+0000 7f4922234640 0 -- 172.21.15.137:0/3304148169 >> v1:172.21.15.137:6800/1497967517 conn(0x7f490404e450 legacy=0x7f490404c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:46:35.249 INFO:teuthology.orchestra.run.smithi137.stdout:2024-09-16T02:46:35.253291+0000 min lat: 0.00196774 max lat: 0.0480783 avg lat: 0.0100298 2024-09-16T02:46:35.249 INFO:teuthology.orchestra.run.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T02:46:35.250 INFO:teuthology.orchestra.run.smithi137.stdout: 20 8 31901 31893 0.00152037 0.00103569 0.0170767 0.0100298 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Total time run: 20.0135 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Total writes made: 31901 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Write size: 1 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Object size: 1 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Bandwidth (MB/sec): 0.00152013 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Stddev Bandwidth: 0.000444233 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Max bandwidth (MB/sec): 0.0025425 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Min bandwidth (MB/sec): 0.00103569 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Average IOPS: 1593 2024-09-16T02:46:36.250 INFO:teuthology.orchestra.run.smithi137.stdout:Stddev IOPS: 465.812 2024-09-16T02:46:36.251 INFO:teuthology.orchestra.run.smithi137.stdout:Max IOPS: 2666 2024-09-16T02:46:36.251 INFO:teuthology.orchestra.run.smithi137.stdout:Min IOPS: 1086 2024-09-16T02:46:36.251 INFO:teuthology.orchestra.run.smithi137.stdout:Average Latency(s): 0.010031 2024-09-16T02:46:36.251 INFO:teuthology.orchestra.run.smithi137.stdout:Stddev Latency(s): 0.00501964 2024-09-16T02:46:36.251 INFO:teuthology.orchestra.run.smithi137.stdout:Max latency(s): 0.0480783 2024-09-16T02:46:36.251 INFO:teuthology.orchestra.run.smithi137.stdout:Min latency(s): 0.00196774 2024-09-16T02:46:36.262 INFO:tasks.osd_recovery:rados -p rbd put foo.1 /etc/passwd 2024-09-16T02:46:36.262 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.1 /etc/passwd 2024-09-16T02:46:36.352 INFO:tasks.osd_recovery:rados -p rbd put foo.2 /etc/passwd 2024-09-16T02:46:36.353 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.2 /etc/passwd 2024-09-16T02:46:36.392 INFO:tasks.osd_recovery:rados -p rbd put foo.3 /etc/passwd 2024-09-16T02:46:36.392 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.3 /etc/passwd 2024-09-16T02:46:36.482 INFO:tasks.osd_recovery:rados -p rbd put foo.4 /etc/passwd 2024-09-16T02:46:36.482 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.4 /etc/passwd 2024-09-16T02:46:36.521 INFO:tasks.osd_recovery:rados -p rbd put foo.5 /etc/passwd 2024-09-16T02:46:36.521 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.5 /etc/passwd 2024-09-16T02:46:36.601 INFO:tasks.osd_recovery:rados -p rbd put foo.6 /etc/passwd 2024-09-16T02:46:36.601 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.6 /etc/passwd 2024-09-16T02:46:36.686 INFO:tasks.osd_recovery:rados -p rbd put foo.7 /etc/passwd 2024-09-16T02:46:36.686 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.7 /etc/passwd 2024-09-16T02:46:36.767 INFO:tasks.osd_recovery:rados -p rbd put foo.8 /etc/passwd 2024-09-16T02:46:36.767 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.8 /etc/passwd 2024-09-16T02:46:36.854 INFO:tasks.osd_recovery:rados -p rbd put foo.9 /etc/passwd 2024-09-16T02:46:36.854 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.9 /etc/passwd 2024-09-16T02:46:36.933 INFO:tasks.osd_recovery:rados -p rbd put foo.10 /etc/passwd 2024-09-16T02:46:36.933 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.10 /etc/passwd 2024-09-16T02:46:37.020 INFO:tasks.osd_recovery:rados -p rbd put foo.11 /etc/passwd 2024-09-16T02:46:37.021 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.11 /etc/passwd 2024-09-16T02:46:37.103 INFO:tasks.osd_recovery:rados -p rbd put foo.12 /etc/passwd 2024-09-16T02:46:37.103 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.12 /etc/passwd 2024-09-16T02:46:37.194 INFO:tasks.osd_recovery:rados -p rbd put foo.13 /etc/passwd 2024-09-16T02:46:37.194 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.13 /etc/passwd 2024-09-16T02:46:37.274 INFO:tasks.osd_recovery:rados -p rbd put foo.14 /etc/passwd 2024-09-16T02:46:37.274 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.14 /etc/passwd 2024-09-16T02:46:37.353 INFO:tasks.osd_recovery:rados -p rbd put foo.15 /etc/passwd 2024-09-16T02:46:37.354 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.15 /etc/passwd 2024-09-16T02:46:37.435 INFO:tasks.osd_recovery:rados -p rbd put foo.16 /etc/passwd 2024-09-16T02:46:37.435 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.16 /etc/passwd 2024-09-16T02:46:37.529 INFO:tasks.osd_recovery:rados -p rbd put foo.17 /etc/passwd 2024-09-16T02:46:37.529 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.17 /etc/passwd 2024-09-16T02:46:37.567 INFO:tasks.osd_recovery:rados -p rbd put foo.18 /etc/passwd 2024-09-16T02:46:37.567 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.18 /etc/passwd 2024-09-16T02:46:37.652 INFO:tasks.osd_recovery:rados -p rbd put foo.19 /etc/passwd 2024-09-16T02:46:37.652 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.19 /etc/passwd 2024-09-16T02:46:37.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 1 2024-09-16T02:46:38.071 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-09-16T02:46:38.068+0000 7f36593c7640 -1 mon.a@0(leader).osd e19 definitely_dead 0 2024-09-16T02:46:38.793 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.0. marked in osd.1. 2024-09-16T02:46:38.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 3 2024-09-16T02:46:39.133 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-09-16T02:46:39.132+0000 7f36593c7640 -1 mon.a@0(leader).osd e20 definitely_dead 0 2024-09-16T02:46:39.813 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.2. marked out osd.3. 2024-09-16T02:46:49.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T02:46:49.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T02:46:49.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T02:46:49.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T02:46:50.036 INFO:teuthology.orchestra.run.smithi137.stdout:42949673178 2024-09-16T02:46:50.037 INFO:teuthology.orchestra.run.smithi137.stdout:38654705882 2024-09-16T02:46:50.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:46:50.038 DEBUG:teuthology.parallel:result is None 2024-09-16T02:46:50.049 INFO:teuthology.orchestra.run.smithi137.stdout:47244640475 2024-09-16T02:46:50.049 DEBUG:teuthology.parallel:result is None 2024-09-16T02:46:50.051 INFO:teuthology.orchestra.run.smithi137.stdout:47244640474 2024-09-16T02:46:50.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:46:50.376 INFO:teuthology.orchestra.run.smithi137.stdout:42949673177 2024-09-16T02:46:50.396 INFO:tasks.osd_recovery.ceph_manager:need seq 42949673178 got 42949673177 for osd.0 2024-09-16T02:46:50.420 INFO:teuthology.orchestra.run.smithi137.stdout:47244640473 2024-09-16T02:46:50.440 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640474 got 47244640473 for osd.1 2024-09-16T02:46:51.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:46:51.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:46:51.752 INFO:teuthology.orchestra.run.smithi137.stdout:42949673177 2024-09-16T02:46:51.776 INFO:tasks.osd_recovery.ceph_manager:need seq 42949673178 got 42949673177 for osd.0 2024-09-16T02:46:51.776 INFO:teuthology.orchestra.run.smithi137.stdout:47244640474 2024-09-16T02:46:51.796 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640474 got 47244640474 for osd.1 2024-09-16T02:46:51.796 DEBUG:teuthology.parallel:result is None 2024-09-16T02:46:52.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:46:53.114 INFO:teuthology.orchestra.run.smithi137.stdout:42949673179 2024-09-16T02:46:53.135 INFO:tasks.osd_recovery.ceph_manager:need seq 42949673178 got 42949673179 for osd.0 2024-09-16T02:46:53.135 DEBUG:teuthology.parallel:result is None 2024-09-16T02:47:03.135 INFO:tasks.osd_recovery.ceph_manager:waiting for peering to complete 2024-09-16T02:47:03.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:03.462 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:03.463 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:03.482 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":556,"stamp":"2024-09-16T02:47:01.651573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34753,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":117396,"kb_used_data":5044,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188626284,"statfs":{"total":193273528320,"available":193153314816,"internally_reserved":0,"allocated":5165056,"data_stored":110741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-736,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003616"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3979,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.075138+0000","last_change":"2024-09-16T02:46:42.075138+0000","last_active":"2024-09-16T02:46:42.075138+0000","last_peered":"2024-09-16T02:46:42.075138+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:46:42.069427+0000","last_became_peered":"2024-09-16T02:46:42.069427+0000","last_unstale":"2024-09-16T02:46:42.075138+0000","last_undegraded":"2024-09-16T02:46:41.658532+0000","last_fullsized":"2024-09-16T02:46:41.319428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4032,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153427+0000","last_change":"2024-09-16T02:46:42.153427+0000","last_active":"2024-09-16T02:46:42.153427+0000","last_peered":"2024-09-16T02:46:42.153427+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:46:42.146294+0000","last_became_peered":"2024-09-16T02:46:42.146294+0000","last_unstale":"2024-09-16T02:46:42.153427+0000","last_undegraded":"2024-09-16T02:46:41.752426+0000","last_fullsized":"2024-09-16T02:46:41.319487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4103,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153351+0000","last_change":"2024-09-16T02:46:42.155410+0000","last_active":"2024-09-16T02:46:42.153351+0000","last_peered":"2024-09-16T02:46:42.153351+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:46:42.147753+0000","last_became_peered":"2024-09-16T02:46:42.147753+0000","last_unstale":"2024-09-16T02:46:42.153351+0000","last_undegraded":"2024-09-16T02:46:41.777529+0000","last_fullsized":"2024-09-16T02:46:41.319546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4004,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.152207+0000","last_change":"2024-09-16T02:46:42.152207+0000","last_active":"2024-09-16T02:46:42.152207+0000","last_peered":"2024-09-16T02:46:42.152207+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:46:42.146107+0000","last_became_peered":"2024-09-16T02:46:42.146107+0000","last_unstale":"2024-09-16T02:46:42.152207+0000","last_undegraded":"2024-09-16T02:46:41.723476+0000","last_fullsized":"2024-09-16T02:46:41.319132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4060,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151922+0000","last_change":"2024-09-16T02:46:42.154138+0000","last_active":"2024-09-16T02:46:42.151922+0000","last_peered":"2024-09-16T02:46:42.151922+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:46:42.146562+0000","last_became_peered":"2024-09-16T02:46:42.146562+0000","last_unstale":"2024-09-16T02:46:42.151922+0000","last_undegraded":"2024-09-16T02:46:41.702503+0000","last_fullsized":"2024-09-16T02:46:41.319025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4029,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.113230+0000","last_change":"2024-09-16T02:46:42.113230+0000","last_active":"2024-09-16T02:46:42.113230+0000","last_peered":"2024-09-16T02:46:42.113230+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:46:42.108737+0000","last_became_peered":"2024-09-16T02:46:42.108737+0000","last_unstale":"2024-09-16T02:46:42.113230+0000","last_undegraded":"2024-09-16T02:46:41.749324+0000","last_fullsized":"2024-09-16T02:46:41.319024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":5579,"reported_epoch":23,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:46:57.642387+0000","last_change":"2024-09-16T02:46:42.071277+0000","last_active":"2024-09-16T02:46:57.642387+0000","last_peered":"2024-09-16T02:46:57.642387+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:46:42.043424+0000","last_became_peered":"2024-09-16T02:46:42.043424+0000","last_unstale":"2024-09-16T02:46:57.642387+0000","last_undegraded":"2024-09-16T02:46:57.642387+0000","last_fullsized":"2024-09-16T02:46:41.318998+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6740,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":540},{"shards":"2,3","objects":3370}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4064,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151127+0000","last_change":"2024-09-16T02:46:42.153507+0000","last_active":"2024-09-16T02:46:42.151127+0000","last_peered":"2024-09-16T02:46:42.151127+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:46:42.146373+0000","last_became_peered":"2024-09-16T02:46:42.146373+0000","last_unstale":"2024-09-16T02:46:42.151127+0000","last_undegraded":"2024-09-16T02:46:41.771833+0000","last_fullsized":"2024-09-16T02:46:41.319001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34753,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266924032,"data_stored":562935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640476,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69560,"kb_used_data":2280,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302280,"statfs":{"total":96636764160,"available":96565534720,"internally_reserved":0,"allocated":2334720,"data_stored":55310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47836,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324004,"statfs":{"total":96636764160,"available":96587780096,"internally_reserved":0,"allocated":2830336,"data_stored":55431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2748416,"data_stored":671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2203648,"data_stored":538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:03.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:03.828 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:03.828 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:03.848 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2024-09-16T02:47:03.652247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":118072,"kb_used_data":5720,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188625608,"statfs":{"total":193273528320,"available":193152622592,"internally_reserved":0,"allocated":5857280,"data_stored":110910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-922,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3979,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.075138+0000","last_change":"2024-09-16T02:46:42.075138+0000","last_active":"2024-09-16T02:46:42.075138+0000","last_peered":"2024-09-16T02:46:42.075138+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:46:42.069427+0000","last_became_peered":"2024-09-16T02:46:42.069427+0000","last_unstale":"2024-09-16T02:46:42.075138+0000","last_undegraded":"2024-09-16T02:46:41.658532+0000","last_fullsized":"2024-09-16T02:46:41.319428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4032,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153427+0000","last_change":"2024-09-16T02:46:42.153427+0000","last_active":"2024-09-16T02:46:42.153427+0000","last_peered":"2024-09-16T02:46:42.153427+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:46:42.146294+0000","last_became_peered":"2024-09-16T02:46:42.146294+0000","last_unstale":"2024-09-16T02:46:42.153427+0000","last_undegraded":"2024-09-16T02:46:41.752426+0000","last_fullsized":"2024-09-16T02:46:41.319487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4103,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153351+0000","last_change":"2024-09-16T02:46:42.155410+0000","last_active":"2024-09-16T02:46:42.153351+0000","last_peered":"2024-09-16T02:46:42.153351+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:46:42.147753+0000","last_became_peered":"2024-09-16T02:46:42.147753+0000","last_unstale":"2024-09-16T02:46:42.153351+0000","last_undegraded":"2024-09-16T02:46:41.777529+0000","last_fullsized":"2024-09-16T02:46:41.319546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4004,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.152207+0000","last_change":"2024-09-16T02:46:42.152207+0000","last_active":"2024-09-16T02:46:42.152207+0000","last_peered":"2024-09-16T02:46:42.152207+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:46:42.146107+0000","last_became_peered":"2024-09-16T02:46:42.146107+0000","last_unstale":"2024-09-16T02:46:42.152207+0000","last_undegraded":"2024-09-16T02:46:41.723476+0000","last_fullsized":"2024-09-16T02:46:41.319132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4060,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151922+0000","last_change":"2024-09-16T02:46:42.154138+0000","last_active":"2024-09-16T02:46:42.151922+0000","last_peered":"2024-09-16T02:46:42.151922+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:46:42.146562+0000","last_became_peered":"2024-09-16T02:46:42.146562+0000","last_unstale":"2024-09-16T02:46:42.151922+0000","last_undegraded":"2024-09-16T02:46:41.702503+0000","last_fullsized":"2024-09-16T02:46:41.319025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4029,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.113230+0000","last_change":"2024-09-16T02:46:42.113230+0000","last_active":"2024-09-16T02:46:42.113230+0000","last_peered":"2024-09-16T02:46:42.113230+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:46:42.108737+0000","last_became_peered":"2024-09-16T02:46:42.108737+0000","last_unstale":"2024-09-16T02:46:42.113230+0000","last_undegraded":"2024-09-16T02:46:41.749324+0000","last_fullsized":"2024-09-16T02:46:41.319024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6092,"reported_epoch":23,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:02.659767+0000","last_change":"2024-09-16T02:46:42.071277+0000","last_active":"2024-09-16T02:47:02.659767+0000","last_peered":"2024-09-16T02:47:02.659767+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:46:42.043424+0000","last_became_peered":"2024-09-16T02:46:42.043424+0000","last_unstale":"2024-09-16T02:47:02.659767+0000","last_undegraded":"2024-09-16T02:47:02.659767+0000","last_fullsized":"2024-09-16T02:46:41.318998+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6398,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":711,"num_bytes_recovered":711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":711},{"shards":"2,3","objects":3199}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4064,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151127+0000","last_change":"2024-09-16T02:46:42.153507+0000","last_active":"2024-09-16T02:46:42.151127+0000","last_peered":"2024-09-16T02:46:42.151127+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:46:42.146373+0000","last_became_peered":"2024-09-16T02:46:42.146373+0000","last_unstale":"2024-09-16T02:46:42.151127+0000","last_undegraded":"2024-09-16T02:46:41.771833+0000","last_fullsized":"2024-09-16T02:46:41.319001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267624448,"data_stored":563106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70236,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301604,"statfs":{"total":96636764160,"available":96564842496,"internally_reserved":0,"allocated":3026944,"data_stored":55479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47836,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324004,"statfs":{"total":96636764160,"available":96587780096,"internally_reserved":0,"allocated":2830336,"data_stored":55431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2748416,"data_stored":671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2904064,"data_stored":709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:03.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:47:04.290 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:04.290 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28013/63846 objects degraded (43.876%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1083,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":23,"num_osds":4,"num_up_osds":4,"osd_up_since":1726453751,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":120905728,"bytes_avail":193152622592,"bytes_total":193273528320,"degraded_objects":28013,"degraded_total":63846,"degraded_ratio":0.43875888857563511,"misplaced_objects":34411,"misplaced_total":63846,"misplaced_ratio":0.53896876859944243,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":38,"recovering_keys_per_sec":0,"num_objects_recovered":461,"num_bytes_recovered":461,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:47:04.308 INFO:tasks.osd_recovery.ceph_manager:{'fsid': 'eabc9450-e776-4be8-8adb-00ce348fa2c2', 'health': {'status': 'HEALTH_WARN', 'checks': {'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 28013/63846 objects degraded (43.876%), 7 pgs degraded', 'count': 7}, 'muted': False}}, 'mutes': []}, 'election_epoch': 4, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 1083, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 23, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1726453751, 'num_in_osds': 2, 'osd_in_since': 1726454799, 'num_remapped_pgs': 8}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_wait+undersized+degraded+remapped', 'count': 7}, {'state_name': 'active+recovering+undersized+remapped', 'count': 1}], 'num_pgs': 8, 'num_pools': 1, 'num_objects': 31923, 'data_bytes': 280863, 'bytes_used': 120905728, 'bytes_avail': 193152622592, 'bytes_total': 193273528320, 'degraded_objects': 28013, 'degraded_total': 63846, 'degraded_ratio': 0.4387588885756351, 'misplaced_objects': 34411, 'misplaced_total': 63846, 'misplaced_ratio': 0.5389687685994424, 'recovering_objects_per_sec': 38, 'recovering_bytes_per_sec': 38, 'recovering_keys_per_sec': 0, 'num_objects_recovered': 461, 'num_bytes_recovered': 461, 'num_keys_recovered': 0}, 'fsmap': {'epoch': 1, 'btime': '2024-09-16T02:29:00:412138+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 9, 'modified': '2024-09-16T02:45:37.631633+0000', 'services': {}}, 'progress_events': {}} 2024-09-16T02:47:04.309 INFO:tasks.osd_recovery.ceph_manager:active! 2024-09-16T02:47:04.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:04.604 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:04.604 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:04.622 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2024-09-16T02:47:03.652247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":118072,"kb_used_data":5720,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188625608,"statfs":{"total":193273528320,"available":193152622592,"internally_reserved":0,"allocated":5857280,"data_stored":110910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-922,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3979,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.075138+0000","last_change":"2024-09-16T02:46:42.075138+0000","last_active":"2024-09-16T02:46:42.075138+0000","last_peered":"2024-09-16T02:46:42.075138+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:46:42.069427+0000","last_became_peered":"2024-09-16T02:46:42.069427+0000","last_unstale":"2024-09-16T02:46:42.075138+0000","last_undegraded":"2024-09-16T02:46:41.658532+0000","last_fullsized":"2024-09-16T02:46:41.319428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4032,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153427+0000","last_change":"2024-09-16T02:46:42.153427+0000","last_active":"2024-09-16T02:46:42.153427+0000","last_peered":"2024-09-16T02:46:42.153427+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:46:42.146294+0000","last_became_peered":"2024-09-16T02:46:42.146294+0000","last_unstale":"2024-09-16T02:46:42.153427+0000","last_undegraded":"2024-09-16T02:46:41.752426+0000","last_fullsized":"2024-09-16T02:46:41.319487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4103,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153351+0000","last_change":"2024-09-16T02:46:42.155410+0000","last_active":"2024-09-16T02:46:42.153351+0000","last_peered":"2024-09-16T02:46:42.153351+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:46:42.147753+0000","last_became_peered":"2024-09-16T02:46:42.147753+0000","last_unstale":"2024-09-16T02:46:42.153351+0000","last_undegraded":"2024-09-16T02:46:41.777529+0000","last_fullsized":"2024-09-16T02:46:41.319546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4004,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.152207+0000","last_change":"2024-09-16T02:46:42.152207+0000","last_active":"2024-09-16T02:46:42.152207+0000","last_peered":"2024-09-16T02:46:42.152207+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:46:42.146107+0000","last_became_peered":"2024-09-16T02:46:42.146107+0000","last_unstale":"2024-09-16T02:46:42.152207+0000","last_undegraded":"2024-09-16T02:46:41.723476+0000","last_fullsized":"2024-09-16T02:46:41.319132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4060,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151922+0000","last_change":"2024-09-16T02:46:42.154138+0000","last_active":"2024-09-16T02:46:42.151922+0000","last_peered":"2024-09-16T02:46:42.151922+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:46:42.146562+0000","last_became_peered":"2024-09-16T02:46:42.146562+0000","last_unstale":"2024-09-16T02:46:42.151922+0000","last_undegraded":"2024-09-16T02:46:41.702503+0000","last_fullsized":"2024-09-16T02:46:41.319025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4029,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.113230+0000","last_change":"2024-09-16T02:46:42.113230+0000","last_active":"2024-09-16T02:46:42.113230+0000","last_peered":"2024-09-16T02:46:42.113230+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:46:42.108737+0000","last_became_peered":"2024-09-16T02:46:42.108737+0000","last_unstale":"2024-09-16T02:46:42.113230+0000","last_undegraded":"2024-09-16T02:46:41.749324+0000","last_fullsized":"2024-09-16T02:46:41.319024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6092,"reported_epoch":23,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:02.659767+0000","last_change":"2024-09-16T02:46:42.071277+0000","last_active":"2024-09-16T02:47:02.659767+0000","last_peered":"2024-09-16T02:47:02.659767+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:46:42.043424+0000","last_became_peered":"2024-09-16T02:46:42.043424+0000","last_unstale":"2024-09-16T02:47:02.659767+0000","last_undegraded":"2024-09-16T02:47:02.659767+0000","last_fullsized":"2024-09-16T02:46:41.318998+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6398,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":711,"num_bytes_recovered":711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":711},{"shards":"2,3","objects":3199}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4064,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151127+0000","last_change":"2024-09-16T02:46:42.153507+0000","last_active":"2024-09-16T02:46:42.151127+0000","last_peered":"2024-09-16T02:46:42.151127+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:46:42.146373+0000","last_became_peered":"2024-09-16T02:46:42.146373+0000","last_unstale":"2024-09-16T02:46:42.151127+0000","last_undegraded":"2024-09-16T02:46:41.771833+0000","last_fullsized":"2024-09-16T02:46:41.319001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267624448,"data_stored":563106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70236,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301604,"statfs":{"total":96636764160,"available":96564842496,"internally_reserved":0,"allocated":3026944,"data_stored":55479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47836,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324004,"statfs":{"total":96636764160,"available":96587780096,"internally_reserved":0,"allocated":2830336,"data_stored":55431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2748416,"data_stored":671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2904064,"data_stored":709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:04.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:04.927 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:04.944 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2024-09-16T02:47:03.652247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":118072,"kb_used_data":5720,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188625608,"statfs":{"total":193273528320,"available":193152622592,"internally_reserved":0,"allocated":5857280,"data_stored":110910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-922,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3979,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.075138+0000","last_change":"2024-09-16T02:46:42.075138+0000","last_active":"2024-09-16T02:46:42.075138+0000","last_peered":"2024-09-16T02:46:42.075138+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:46:42.069427+0000","last_became_peered":"2024-09-16T02:46:42.069427+0000","last_unstale":"2024-09-16T02:46:42.075138+0000","last_undegraded":"2024-09-16T02:46:41.658532+0000","last_fullsized":"2024-09-16T02:46:41.319428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4032,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153427+0000","last_change":"2024-09-16T02:46:42.153427+0000","last_active":"2024-09-16T02:46:42.153427+0000","last_peered":"2024-09-16T02:46:42.153427+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:46:42.146294+0000","last_became_peered":"2024-09-16T02:46:42.146294+0000","last_unstale":"2024-09-16T02:46:42.153427+0000","last_undegraded":"2024-09-16T02:46:41.752426+0000","last_fullsized":"2024-09-16T02:46:41.319487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4103,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153351+0000","last_change":"2024-09-16T02:46:42.155410+0000","last_active":"2024-09-16T02:46:42.153351+0000","last_peered":"2024-09-16T02:46:42.153351+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:46:42.147753+0000","last_became_peered":"2024-09-16T02:46:42.147753+0000","last_unstale":"2024-09-16T02:46:42.153351+0000","last_undegraded":"2024-09-16T02:46:41.777529+0000","last_fullsized":"2024-09-16T02:46:41.319546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4004,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.152207+0000","last_change":"2024-09-16T02:46:42.152207+0000","last_active":"2024-09-16T02:46:42.152207+0000","last_peered":"2024-09-16T02:46:42.152207+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:46:42.146107+0000","last_became_peered":"2024-09-16T02:46:42.146107+0000","last_unstale":"2024-09-16T02:46:42.152207+0000","last_undegraded":"2024-09-16T02:46:41.723476+0000","last_fullsized":"2024-09-16T02:46:41.319132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4060,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151922+0000","last_change":"2024-09-16T02:46:42.154138+0000","last_active":"2024-09-16T02:46:42.151922+0000","last_peered":"2024-09-16T02:46:42.151922+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:46:42.146562+0000","last_became_peered":"2024-09-16T02:46:42.146562+0000","last_unstale":"2024-09-16T02:46:42.151922+0000","last_undegraded":"2024-09-16T02:46:41.702503+0000","last_fullsized":"2024-09-16T02:46:41.319025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4029,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.113230+0000","last_change":"2024-09-16T02:46:42.113230+0000","last_active":"2024-09-16T02:46:42.113230+0000","last_peered":"2024-09-16T02:46:42.113230+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:46:42.108737+0000","last_became_peered":"2024-09-16T02:46:42.108737+0000","last_unstale":"2024-09-16T02:46:42.113230+0000","last_undegraded":"2024-09-16T02:46:41.749324+0000","last_fullsized":"2024-09-16T02:46:41.319024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6092,"reported_epoch":23,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:02.659767+0000","last_change":"2024-09-16T02:46:42.071277+0000","last_active":"2024-09-16T02:47:02.659767+0000","last_peered":"2024-09-16T02:47:02.659767+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:46:42.043424+0000","last_became_peered":"2024-09-16T02:46:42.043424+0000","last_unstale":"2024-09-16T02:47:02.659767+0000","last_undegraded":"2024-09-16T02:47:02.659767+0000","last_fullsized":"2024-09-16T02:46:41.318998+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6398,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":711,"num_bytes_recovered":711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":711},{"shards":"2,3","objects":3199}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4064,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151127+0000","last_change":"2024-09-16T02:46:42.153507+0000","last_active":"2024-09-16T02:46:42.151127+0000","last_peered":"2024-09-16T02:46:42.151127+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:46:42.146373+0000","last_became_peered":"2024-09-16T02:46:42.146373+0000","last_unstale":"2024-09-16T02:46:42.151127+0000","last_undegraded":"2024-09-16T02:46:41.771833+0000","last_fullsized":"2024-09-16T02:46:41.319001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267624448,"data_stored":563106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70236,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301604,"statfs":{"total":96636764160,"available":96564842496,"internally_reserved":0,"allocated":3026944,"data_stored":55479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47836,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324004,"statfs":{"total":96636764160,"available":96587780096,"internally_reserved":0,"allocated":2830336,"data_stored":55431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2748416,"data_stored":671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2904064,"data_stored":709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:04.945 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:47:04.945 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3979, 'reported_epoch': 23, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:46:42.075138+0000', 'last_change': '2024-09-16T02:46:42.075138+0000', 'last_active': '2024-09-16T02:46:42.075138+0000', 'last_peered': '2024-09-16T02:46:42.075138+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:46:42.069427+0000', 'last_became_peered': '2024-09-16T02:46:42.069427+0000', 'last_unstale': '2024-09-16T02:46:42.075138+0000', 'last_undegraded': '2024-09-16T02:46:41.658532+0000', 'last_fullsized': '2024-09-16T02:46:41.319428+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.945 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:47:04.945 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 4032, 'reported_epoch': 23, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:46:42.153427+0000', 'last_change': '2024-09-16T02:46:42.153427+0000', 'last_active': '2024-09-16T02:46:42.153427+0000', 'last_peered': '2024-09-16T02:46:42.153427+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:46:42.146294+0000', 'last_became_peered': '2024-09-16T02:46:42.146294+0000', 'last_unstale': '2024-09-16T02:46:42.153427+0000', 'last_undegraded': '2024-09-16T02:46:41.752426+0000', 'last_fullsized': '2024-09-16T02:46:41.319487+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3985, 'num_objects_misplaced': 3985, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3985}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.945 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:47:04.945 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4103, 'reported_epoch': 23, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:46:42.153351+0000', 'last_change': '2024-09-16T02:46:42.155410+0000', 'last_active': '2024-09-16T02:46:42.153351+0000', 'last_peered': '2024-09-16T02:46:42.153351+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:46:42.147753+0000', 'last_became_peered': '2024-09-16T02:46:42.147753+0000', 'last_unstale': '2024-09-16T02:46:42.153351+0000', 'last_undegraded': '2024-09-16T02:46:41.777529+0000', 'last_fullsized': '2024-09-16T02:46:41.319546+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4004, 'reported_epoch': 23, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:46:42.152207+0000', 'last_change': '2024-09-16T02:46:42.152207+0000', 'last_active': '2024-09-16T02:46:42.152207+0000', 'last_peered': '2024-09-16T02:46:42.152207+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:46:42.146107+0000', 'last_became_peered': '2024-09-16T02:46:42.146107+0000', 'last_unstale': '2024-09-16T02:46:42.152207+0000', 'last_undegraded': '2024-09-16T02:46:41.723476+0000', 'last_fullsized': '2024-09-16T02:46:41.319132+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4060, 'reported_epoch': 23, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:46:42.151922+0000', 'last_change': '2024-09-16T02:46:42.154138+0000', 'last_active': '2024-09-16T02:46:42.151922+0000', 'last_peered': '2024-09-16T02:46:42.151922+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:46:42.146562+0000', 'last_became_peered': '2024-09-16T02:46:42.146562+0000', 'last_unstale': '2024-09-16T02:46:42.151922+0000', 'last_undegraded': '2024-09-16T02:46:41.702503+0000', 'last_fullsized': '2024-09-16T02:46:41.319025+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4029, 'reported_epoch': 23, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:46:42.113230+0000', 'last_change': '2024-09-16T02:46:42.113230+0000', 'last_active': '2024-09-16T02:46:42.113230+0000', 'last_peered': '2024-09-16T02:46:42.113230+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:46:42.108737+0000', 'last_became_peered': '2024-09-16T02:46:42.108737+0000', 'last_unstale': '2024-09-16T02:46:42.113230+0000', 'last_undegraded': '2024-09-16T02:46:41.749324+0000', 'last_fullsized': '2024-09-16T02:46:41.319024+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "19'3910", 'reported_seq': 6092, 'reported_epoch': 23, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:47:02.659767+0000', 'last_change': '2024-09-16T02:46:42.071277+0000', 'last_active': '2024-09-16T02:47:02.659767+0000', 'last_peered': '2024-09-16T02:47:02.659767+0000', 'last_clean': '2024-09-16T02:46:35.256073+0000', 'last_became_active': '2024-09-16T02:46:42.043424+0000', 'last_became_peered': '2024-09-16T02:46:42.043424+0000', 'last_unstale': '2024-09-16T02:47:02.659767+0000', 'last_undegraded': '2024-09-16T02:47:02.659767+0000', 'last_fullsized': '2024-09-16T02:46:41.318998+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3910, 'log_dups_size': 0, 'ondisk_log_size': 3910, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001711783, 'stat_sum': {'num_bytes': 17010, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6398, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3910, 'num_write_kb': 3922, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 711, 'num_bytes_recovered': 711, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 711}, {'shards': '2,3', 'objects': 3199}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.946 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:47:04.947 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4064, 'reported_epoch': 23, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:46:42.151127+0000', 'last_change': '2024-09-16T02:46:42.153507+0000', 'last_active': '2024-09-16T02:46:42.151127+0000', 'last_peered': '2024-09-16T02:46:42.151127+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:46:42.146373+0000', 'last_became_peered': '2024-09-16T02:46:42.146373+0000', 'last_unstale': '2024-09-16T02:46:42.151127+0000', 'last_undegraded': '2024-09-16T02:46:41.771833+0000', 'last_fullsized': '2024-09-16T02:46:41.319001+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:04.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:05.288 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:05.288 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:05.306 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2024-09-16T02:47:03.652247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":118072,"kb_used_data":5720,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188625608,"statfs":{"total":193273528320,"available":193152622592,"internally_reserved":0,"allocated":5857280,"data_stored":110910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-922,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3979,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.075138+0000","last_change":"2024-09-16T02:46:42.075138+0000","last_active":"2024-09-16T02:46:42.075138+0000","last_peered":"2024-09-16T02:46:42.075138+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:46:42.069427+0000","last_became_peered":"2024-09-16T02:46:42.069427+0000","last_unstale":"2024-09-16T02:46:42.075138+0000","last_undegraded":"2024-09-16T02:46:41.658532+0000","last_fullsized":"2024-09-16T02:46:41.319428+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4032,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153427+0000","last_change":"2024-09-16T02:46:42.153427+0000","last_active":"2024-09-16T02:46:42.153427+0000","last_peered":"2024-09-16T02:46:42.153427+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:46:42.146294+0000","last_became_peered":"2024-09-16T02:46:42.146294+0000","last_unstale":"2024-09-16T02:46:42.153427+0000","last_undegraded":"2024-09-16T02:46:41.752426+0000","last_fullsized":"2024-09-16T02:46:41.319487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4103,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.153351+0000","last_change":"2024-09-16T02:46:42.155410+0000","last_active":"2024-09-16T02:46:42.153351+0000","last_peered":"2024-09-16T02:46:42.153351+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:46:42.147753+0000","last_became_peered":"2024-09-16T02:46:42.147753+0000","last_unstale":"2024-09-16T02:46:42.153351+0000","last_undegraded":"2024-09-16T02:46:41.777529+0000","last_fullsized":"2024-09-16T02:46:41.319546+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4004,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.152207+0000","last_change":"2024-09-16T02:46:42.152207+0000","last_active":"2024-09-16T02:46:42.152207+0000","last_peered":"2024-09-16T02:46:42.152207+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:46:42.146107+0000","last_became_peered":"2024-09-16T02:46:42.146107+0000","last_unstale":"2024-09-16T02:46:42.152207+0000","last_undegraded":"2024-09-16T02:46:41.723476+0000","last_fullsized":"2024-09-16T02:46:41.319132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4060,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151922+0000","last_change":"2024-09-16T02:46:42.154138+0000","last_active":"2024-09-16T02:46:42.151922+0000","last_peered":"2024-09-16T02:46:42.151922+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:46:42.146562+0000","last_became_peered":"2024-09-16T02:46:42.146562+0000","last_unstale":"2024-09-16T02:46:42.151922+0000","last_undegraded":"2024-09-16T02:46:41.702503+0000","last_fullsized":"2024-09-16T02:46:41.319025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4029,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.113230+0000","last_change":"2024-09-16T02:46:42.113230+0000","last_active":"2024-09-16T02:46:42.113230+0000","last_peered":"2024-09-16T02:46:42.113230+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:46:42.108737+0000","last_became_peered":"2024-09-16T02:46:42.108737+0000","last_unstale":"2024-09-16T02:46:42.113230+0000","last_undegraded":"2024-09-16T02:46:41.749324+0000","last_fullsized":"2024-09-16T02:46:41.319024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6092,"reported_epoch":23,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:02.659767+0000","last_change":"2024-09-16T02:46:42.071277+0000","last_active":"2024-09-16T02:47:02.659767+0000","last_peered":"2024-09-16T02:47:02.659767+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:46:42.043424+0000","last_became_peered":"2024-09-16T02:46:42.043424+0000","last_unstale":"2024-09-16T02:47:02.659767+0000","last_undegraded":"2024-09-16T02:47:02.659767+0000","last_fullsized":"2024-09-16T02:46:41.318998+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6398,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":711,"num_bytes_recovered":711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":711},{"shards":"2,3","objects":3199}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4064,"reported_epoch":23,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:46:42.151127+0000","last_change":"2024-09-16T02:46:42.153507+0000","last_active":"2024-09-16T02:46:42.151127+0000","last_peered":"2024-09-16T02:46:42.151127+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:46:42.146373+0000","last_became_peered":"2024-09-16T02:46:42.146373+0000","last_unstale":"2024-09-16T02:46:42.151127+0000","last_undegraded":"2024-09-16T02:46:41.771833+0000","last_fullsized":"2024-09-16T02:46:41.319001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":34411,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267624448,"data_stored":563106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70236,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301604,"statfs":{"total":96636764160,"available":96564842496,"internally_reserved":0,"allocated":3026944,"data_stored":55479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673181,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47836,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324004,"statfs":{"total":96636764160,"available":96587780096,"internally_reserved":0,"allocated":2830336,"data_stored":55431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2748416,"data_stored":671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2904064,"data_stored":709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:05.307 INFO:tasks.osd_recovery:stopping 2,3 2024-09-16T02:47:05.307 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T02:47:05.307 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:47:08.381 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-16T02:47:11.409 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T02:47:11.409 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T02:47:11.410 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:47:11.494 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T02:47:11.494 INFO:tasks.osd_recovery:... 2024-09-16T02:47:11.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 3 2024-09-16T02:47:11.816 INFO:tasks.ceph.mon.a.smithi137.stderr:2024-09-16T02:47:11.815+0000 7f36593c7640 -1 mon.a@0(leader).osd e23 definitely_dead 0 2024-09-16T02:47:12.751 INFO:teuthology.orchestra.run.smithi137.stderr:marked down osd.2. marked down osd.3. 2024-09-16T02:47:12.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T02:47:12.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T02:47:12.941 INFO:teuthology.orchestra.run.smithi137.stdout:47244640480 2024-09-16T02:47:12.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:47:12.945 INFO:teuthology.orchestra.run.smithi137.stdout:42949673184 2024-09-16T02:47:12.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:47:13.237 INFO:teuthology.orchestra.run.smithi137.stdout:47244640478 2024-09-16T02:47:13.255 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640480 got 47244640478 for osd.1 2024-09-16T02:47:13.281 INFO:teuthology.orchestra.run.smithi137.stdout:42949673183 2024-09-16T02:47:13.299 INFO:tasks.osd_recovery.ceph_manager:need seq 42949673184 got 42949673183 for osd.0 2024-09-16T02:47:13.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-16T02:47:14.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:47:14.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:47:14.579 INFO:teuthology.orchestra.run.smithi137.stdout:47244640480 2024-09-16T02:47:14.598 INFO:tasks.osd_recovery.ceph_manager:need seq 47244640480 got 47244640480 for osd.1 2024-09-16T02:47:14.599 DEBUG:teuthology.parallel:result is None 2024-09-16T02:47:14.606 INFO:teuthology.orchestra.run.smithi137.stdout:42949673184 2024-09-16T02:47:14.625 INFO:tasks.osd_recovery.ceph_manager:need seq 42949673184 got 42949673184 for osd.0 2024-09-16T02:47:14.625 DEBUG:teuthology.parallel:result is None 2024-09-16T02:47:14.625 INFO:tasks.osd_recovery.ceph_manager:waiting for peering to complete or become blocked 2024-09-16T02:47:14.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:14.971 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:14.971 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:14.976 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-09-16T02:47:13.751010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":123880,"kb_used_data":7368,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188619800,"statfs":{"total":193273528320,"available":193146675200,"internally_reserved":0,"allocated":7544832,"data_stored":114130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":-28013,"num_objects_misplaced":-34411,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-711,"num_bytes_recovered":-711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.098225"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3975,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.755004+0000","last_change":"2024-09-16T02:47:12.755004+0000","last_active":"2024-09-16T02:46:40.324378+0000","last_peered":"2024-09-16T02:46:35.253185+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:47:12.755004+0000","last_undegraded":"2024-09-16T02:47:12.755004+0000","last_fullsized":"2024-09-16T02:47:12.755004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":3954,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4028,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752674+0000","last_change":"2024-09-16T02:47:12.752674+0000","last_active":"2024-09-16T02:46:40.326018+0000","last_peered":"2024-09-16T02:46:36.687660+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:47:12.752674+0000","last_undegraded":"2024-09-16T02:47:12.752674+0000","last_fullsized":"2024-09-16T02:47:12.752674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":3985,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4100,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752346+0000","last_change":"2024-09-16T02:47:12.752346+0000","last_active":"2024-09-16T02:46:40.324819+0000","last_peered":"2024-09-16T02:46:39.818003+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:47:12.752346+0000","last_undegraded":"2024-09-16T02:47:12.752346+0000","last_fullsized":"2024-09-16T02:47:12.752346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":4060,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4000,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.753375+0000","last_change":"2024-09-16T02:47:12.753375+0000","last_active":"2024-09-16T02:46:40.321755+0000","last_peered":"2024-09-16T02:46:37.354625+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:47:12.753375+0000","last_undegraded":"2024-09-16T02:47:12.753375+0000","last_fullsized":"2024-09-16T02:47:12.753375+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":3956,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4057,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752294+0000","last_change":"2024-09-16T02:47:12.752294+0000","last_active":"2024-09-16T02:46:40.327722+0000","last_peered":"2024-09-16T02:46:36.933938+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:47:12.752294+0000","last_undegraded":"2024-09-16T02:47:12.752294+0000","last_fullsized":"2024-09-16T02:47:12.752294+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":4024,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4025,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.755304+0000","last_change":"2024-09-16T02:47:12.755304+0000","last_active":"2024-09-16T02:46:40.322198+0000","last_peered":"2024-09-16T02:46:37.435854+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:47:12.755304+0000","last_undegraded":"2024-09-16T02:47:12.755304+0000","last_fullsized":"2024-09-16T02:47:12.755304+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":3997,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3954,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.753471+0000","last_change":"2024-09-16T02:47:12.753471+0000","last_active":"2024-09-16T02:46:40.322333+0000","last_peered":"2024-09-16T02:46:35.256073+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:47:12.753471+0000","last_undegraded":"2024-09-16T02:47:12.753471+0000","last_fullsized":"2024-09-16T02:47:12.753471+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":3038,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4061,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752400+0000","last_change":"2024-09-16T02:47:12.752400+0000","last_active":"2024-09-16T02:46:40.327946+0000","last_peered":"2024-09-16T02:46:37.653464+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:47:12.752400+0000","last_undegraded":"2024-09-16T02:47:12.752400+0000","last_fullsized":"2024-09-16T02:47:12.752400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":4037,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269115392,"data_stored":563470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640480,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70960,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300880,"statfs":{"total":96636764160,"available":96564101120,"internally_reserved":0,"allocated":3768320,"data_stored":55660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673184,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52920,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318920,"statfs":{"total":96636764160,"available":96582574080,"internally_reserved":0,"allocated":3776512,"data_stored":58470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3571712,"data_stored":872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3571712,"data_stored":872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:14.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:15.260 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:15.261 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:15.276 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-09-16T02:47:13.751010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":123880,"kb_used_data":7368,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188619800,"statfs":{"total":193273528320,"available":193146675200,"internally_reserved":0,"allocated":7544832,"data_stored":114130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":-28013,"num_objects_misplaced":-34411,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-711,"num_bytes_recovered":-711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.098225"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3975,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.755004+0000","last_change":"2024-09-16T02:47:12.755004+0000","last_active":"2024-09-16T02:46:40.324378+0000","last_peered":"2024-09-16T02:46:35.253185+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:47:12.755004+0000","last_undegraded":"2024-09-16T02:47:12.755004+0000","last_fullsized":"2024-09-16T02:47:12.755004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":3954,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4028,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752674+0000","last_change":"2024-09-16T02:47:12.752674+0000","last_active":"2024-09-16T02:46:40.326018+0000","last_peered":"2024-09-16T02:46:36.687660+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:47:12.752674+0000","last_undegraded":"2024-09-16T02:47:12.752674+0000","last_fullsized":"2024-09-16T02:47:12.752674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":3985,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4100,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752346+0000","last_change":"2024-09-16T02:47:12.752346+0000","last_active":"2024-09-16T02:46:40.324819+0000","last_peered":"2024-09-16T02:46:39.818003+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:47:12.752346+0000","last_undegraded":"2024-09-16T02:47:12.752346+0000","last_fullsized":"2024-09-16T02:47:12.752346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":4060,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4000,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.753375+0000","last_change":"2024-09-16T02:47:12.753375+0000","last_active":"2024-09-16T02:46:40.321755+0000","last_peered":"2024-09-16T02:46:37.354625+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:47:12.753375+0000","last_undegraded":"2024-09-16T02:47:12.753375+0000","last_fullsized":"2024-09-16T02:47:12.753375+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":3956,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4057,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752294+0000","last_change":"2024-09-16T02:47:12.752294+0000","last_active":"2024-09-16T02:46:40.327722+0000","last_peered":"2024-09-16T02:46:36.933938+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:47:12.752294+0000","last_undegraded":"2024-09-16T02:47:12.752294+0000","last_fullsized":"2024-09-16T02:47:12.752294+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":4024,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4025,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.755304+0000","last_change":"2024-09-16T02:47:12.755304+0000","last_active":"2024-09-16T02:46:40.322198+0000","last_peered":"2024-09-16T02:46:37.435854+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:47:12.755304+0000","last_undegraded":"2024-09-16T02:47:12.755304+0000","last_fullsized":"2024-09-16T02:47:12.755304+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":3997,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3954,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.753471+0000","last_change":"2024-09-16T02:47:12.753471+0000","last_active":"2024-09-16T02:46:40.322333+0000","last_peered":"2024-09-16T02:46:35.256073+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:47:12.753471+0000","last_undegraded":"2024-09-16T02:47:12.753471+0000","last_fullsized":"2024-09-16T02:47:12.753471+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":3038,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4061,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752400+0000","last_change":"2024-09-16T02:47:12.752400+0000","last_active":"2024-09-16T02:46:40.327946+0000","last_peered":"2024-09-16T02:46:37.653464+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:47:12.752400+0000","last_undegraded":"2024-09-16T02:47:12.752400+0000","last_fullsized":"2024-09-16T02:47:12.752400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":4037,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269115392,"data_stored":563470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640480,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70960,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300880,"statfs":{"total":96636764160,"available":96564101120,"internally_reserved":0,"allocated":3768320,"data_stored":55660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673184,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52920,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318920,"statfs":{"total":96636764160,"available":96582574080,"internally_reserved":0,"allocated":3776512,"data_stored":58470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3571712,"data_stored":872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3571712,"data_stored":872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:15.277 INFO:tasks.osd_recovery.ceph_manager:active or down! 2024-09-16T02:47:15.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:15.558 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:15.559 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:15.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-09-16T02:47:13.751010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":123880,"kb_used_data":7368,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188619800,"statfs":{"total":193273528320,"available":193146675200,"internally_reserved":0,"allocated":7544832,"data_stored":114130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":-28013,"num_objects_misplaced":-34411,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-711,"num_bytes_recovered":-711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.098225"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3975,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.755004+0000","last_change":"2024-09-16T02:47:12.755004+0000","last_active":"2024-09-16T02:46:40.324378+0000","last_peered":"2024-09-16T02:46:35.253185+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:29:29.549079+0000","last_became_peered":"2024-09-16T02:29:29.549079+0000","last_unstale":"2024-09-16T02:47:12.755004+0000","last_undegraded":"2024-09-16T02:47:12.755004+0000","last_fullsized":"2024-09-16T02:47:12.755004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":3954,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4028,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752674+0000","last_change":"2024-09-16T02:47:12.752674+0000","last_active":"2024-09-16T02:46:40.326018+0000","last_peered":"2024-09-16T02:46:36.687660+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:29:29.553499+0000","last_became_peered":"2024-09-16T02:29:29.553499+0000","last_unstale":"2024-09-16T02:47:12.752674+0000","last_undegraded":"2024-09-16T02:47:12.752674+0000","last_fullsized":"2024-09-16T02:47:12.752674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":3985,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4100,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752346+0000","last_change":"2024-09-16T02:47:12.752346+0000","last_active":"2024-09-16T02:46:40.324819+0000","last_peered":"2024-09-16T02:46:39.818003+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:29:17.490594+0000","last_became_peered":"2024-09-16T02:29:17.490594+0000","last_unstale":"2024-09-16T02:47:12.752346+0000","last_undegraded":"2024-09-16T02:47:12.752346+0000","last_fullsized":"2024-09-16T02:47:12.752346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":4060,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4000,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.753375+0000","last_change":"2024-09-16T02:47:12.753375+0000","last_active":"2024-09-16T02:46:40.321755+0000","last_peered":"2024-09-16T02:46:37.354625+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:29:29.549331+0000","last_became_peered":"2024-09-16T02:29:29.549331+0000","last_unstale":"2024-09-16T02:47:12.753375+0000","last_undegraded":"2024-09-16T02:47:12.753375+0000","last_fullsized":"2024-09-16T02:47:12.753375+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":3956,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4057,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752294+0000","last_change":"2024-09-16T02:47:12.752294+0000","last_active":"2024-09-16T02:46:40.327722+0000","last_peered":"2024-09-16T02:46:36.933938+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:29:29.546635+0000","last_became_peered":"2024-09-16T02:29:29.546635+0000","last_unstale":"2024-09-16T02:47:12.752294+0000","last_undegraded":"2024-09-16T02:47:12.752294+0000","last_fullsized":"2024-09-16T02:47:12.752294+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":4024,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4025,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.755304+0000","last_change":"2024-09-16T02:47:12.755304+0000","last_active":"2024-09-16T02:46:40.322198+0000","last_peered":"2024-09-16T02:46:37.435854+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:29:29.553972+0000","last_became_peered":"2024-09-16T02:29:29.553972+0000","last_unstale":"2024-09-16T02:47:12.755304+0000","last_undegraded":"2024-09-16T02:47:12.755304+0000","last_fullsized":"2024-09-16T02:47:12.755304+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":3997,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3954,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.753471+0000","last_change":"2024-09-16T02:47:12.753471+0000","last_active":"2024-09-16T02:46:40.322333+0000","last_peered":"2024-09-16T02:46:35.256073+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:29:29.553380+0000","last_became_peered":"2024-09-16T02:29:29.553380+0000","last_unstale":"2024-09-16T02:47:12.753471+0000","last_undegraded":"2024-09-16T02:47:12.753471+0000","last_fullsized":"2024-09-16T02:47:12.753471+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":3038,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4061,"reported_epoch":24,"state":"down","last_fresh":"2024-09-16T02:47:12.752400+0000","last_change":"2024-09-16T02:47:12.752400+0000","last_active":"2024-09-16T02:46:40.327946+0000","last_peered":"2024-09-16T02:46:37.653464+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:29:29.545723+0000","last_became_peered":"2024-09-16T02:29:29.545723+0000","last_unstale":"2024-09-16T02:47:12.752400+0000","last_undegraded":"2024-09-16T02:47:12.752400+0000","last_fullsized":"2024-09-16T02:47:12.752400+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":4037,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":31051,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269115392,"data_stored":563470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":47244640478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640480,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70960,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300880,"statfs":{"total":96636764160,"available":96564101120,"internally_reserved":0,"allocated":3768320,"data_stored":55660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673184,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52920,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318920,"statfs":{"total":96636764160,"available":96582574080,"internally_reserved":0,"allocated":3776512,"data_stored":58470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":0,"seq":38654705887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3571712,"data_stored":872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3571712,"data_stored":872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:15.575 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T02:47:15.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T02:47:15.577 INFO:tasks.ceph.osd.2:Started 2024-09-16T02:47:15.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:47:15.685 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-16T02:47:15.685 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-16T02:47:15.685 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-16T02:47:15.685 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-16T02:47:15.686 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-16T02:47:15.686 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-16T02:47:15.686 INFO:teuthology.orchestra.run.smithi137.stderr:config diff 2024-09-16T02:47:15.686 INFO:teuthology.orchestra.run.smithi137.stderr:config diff get 2024-09-16T02:47:15.687 INFO:teuthology.orchestra.run.smithi137.stderr:config get 2024-09-16T02:47:15.687 INFO:teuthology.orchestra.run.smithi137.stderr:config help [] 2024-09-16T02:47:15.687 INFO:teuthology.orchestra.run.smithi137.stderr:config set ... 2024-09-16T02:47:15.687 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-09-16T02:47:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T02:47:15.689 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-16T02:47:16.921 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-16T02:47:16.919+0000 7fc3f786a7c0 -1 Falling back to public interface 2024-09-16T02:47:20.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-16T02:47:20.815 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator fragmentation block 2024-09-16T02:47:20.816 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-09-16T02:47:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T02:47:20.818 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-16T02:47:23.365 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-16T02:47:23.363+0000 7fc3f786a7c0 -1 osd.2 23 log_to_monitors true 2024-09-16T02:47:23.792 INFO:tasks.ceph.osd.2.smithi137.stderr:2024-09-16T02:47:23.791+0000 7fc3ef64d640 -1 osd.2 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:47:25.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:47:25.954 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:47:25.954 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-16T02:47:25.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.954 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.4171.0:35 1.2 1.7c339972 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e28)", 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-16T02:47:25.335721+0000", 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.62164038099999996, 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.62168317299999998, 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for peered", 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.4171", 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1604936723", 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 35 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-16T02:47:25.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335721+0000", 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335721+0000", 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335726+0000", 2024-09-16T02:47:25.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0000000000000004e-06 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335744+0000", 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7983e-05 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335772+0000", 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8255999999999998e-05 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335799+0000", 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7169e-05 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335950+0000", 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015034900000000001 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335965+0000", 2024-09-16T02:47:25.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5685e-05 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.4171.0:36 1.1 1.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e28)", 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-16T02:47:25.335846+0000", 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.62151529999999999, 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.62166312700000004, 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for peered", 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-16T02:47:25.959 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.4171", 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1604936723", 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 36 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335846+0000", 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335846+0000", 2024-09-16T02:47:25.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335848+0000", 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.04e-06 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335858+0000", 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7059999999999999e-06 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335876+0000", 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8593e-05 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.335893+0000", 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6750000000000001e-05 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.336143+0000", 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024971500000000002 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-16T02:47:25.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-16T02:47:25.336151+0000", 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4679999999999992e-06 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 2 2024-09-16T02:47:25.963 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:47:25.971 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T02:47:25.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T02:47:25.976 INFO:tasks.ceph.osd.3:Started 2024-09-16T02:47:25.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:config diff 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:config diff get 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:config get 2024-09-16T02:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:config help [] 2024-09-16T02:47:26.101 INFO:teuthology.orchestra.run.smithi137.stderr:config set ... 2024-09-16T02:47:26.101 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-09-16T02:47:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T02:47:26.103 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-16T02:47:27.577 INFO:tasks.ceph.osd.3.smithi137.stderr:2024-09-16T02:47:27.575+0000 7f49519af7c0 -1 Falling back to public interface 2024-09-16T02:47:31.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:47:31.229 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-16T02:47:31.229 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-16T02:47:31.229 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-16T02:47:31.229 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-16T02:47:31.229 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-16T02:47:31.230 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-16T02:47:31.230 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-16T02:47:31.230 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-16T02:47:31.230 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-16T02:47:31.230 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore bluefs device info [] 2024-09-16T02:47:31.230 INFO:teuthology.orchestra.run.smithi137.stderr:config diff 2024-09-16T02:47:31.230 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-09-16T02:47:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T02:47:31.232 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-16T02:47:35.051 INFO:tasks.ceph.osd.3.smithi137.stderr:2024-09-16T02:47:35.051+0000 7f49519af7c0 -1 osd.3 23 log_to_monitors true 2024-09-16T02:47:36.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-16T02:47:36.376 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator fragmentation block 2024-09-16T02:47:36.377 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-09-16T02:47:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T02:47:36.379 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-16T02:47:36.692 INFO:tasks.ceph.osd.3.smithi137.stderr:2024-09-16T02:47:36.691+0000 7f494578a640 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:47:41.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:47:41.533 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:47:41.533 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-16T02:47:41.533 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-16T02:47:41.533 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:47:41.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 31 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:fsid eabc9450-e776-4be8-8adb-00ce348fa2c2 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-09-16T02:29:00.413885+0000 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:modified 2024-09-16T02:47:37.699678+0000 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:crush_version 7 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:full_ratio 0.95 2024-09-16T02:47:41.880 INFO:teuthology.orchestra.run.smithi137.stdout:backfillfull_ratio 0.9 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:nearfull_ratio 0.85 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:require_min_compat_client luminous 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:min_compat_client jewel 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:require_osd_release squid 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:stretch_mode_enabled false 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 16 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.00 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:max_osd 4 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 up in weight 1 up_from 10 up_thru 27 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6804/683366439 v1:172.21.15.137:6805/683366439 exists,up 12c8d8a5-d99d-4bfc-99dd-7b19e351ed44 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 up in weight 1 up_from 11 up_thru 27 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6812/4172164026 v1:172.21.15.137:6813/4172164026 exists,up 8a607138-b9f7-49d3-941d-ca065ebe247f 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 up out weight 0 up_from 27 up_thru 28 down_at 24 last_clean_interval [11,23) v1:172.21.15.137:6800/4262355089 v1:172.21.15.137:6801/4262355089 exists,up 6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 up out weight 0 up_from 31 up_thru 18 down_at 24 last_clean_interval [9,23) v1:172.21.15.137:6808/1833792554 v1:172.21.15.137:6809/1833792554 exists,up 54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.0 [2] 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.1 [2] 2024-09-16T02:47:41.881 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.2 [2] 2024-09-16T02:47:41.882 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.3 [2] 2024-09-16T02:47:41.882 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.4 [2] 2024-09-16T02:47:41.882 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.5 [2] 2024-09-16T02:47:41.882 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.6 [2] 2024-09-16T02:47:41.882 INFO:teuthology.orchestra.run.smithi137.stdout:pg_temp 1.7 [2] 2024-09-16T02:47:41.900 INFO:tasks.osd_recovery.ceph_manager:['osd.0 up in weight 1 up_from 10 up_thru 27 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6804/683366439 v1:172.21.15.137:6805/683366439 exists,up 12c8d8a5-d99d-4bfc-99dd-7b19e351ed44', 'osd.1 up in weight 1 up_from 11 up_thru 27 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6812/4172164026 v1:172.21.15.137:6813/4172164026 exists,up 8a607138-b9f7-49d3-941d-ca065ebe247f', 'osd.2 up out weight 0 up_from 27 up_thru 28 down_at 24 last_clean_interval [11,23) v1:172.21.15.137:6800/4262355089 v1:172.21.15.137:6801/4262355089 exists,up 6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82', 'osd.3 up out weight 0 up_from 31 up_thru 18 down_at 24 last_clean_interval [9,23) v1:172.21.15.137:6808/1833792554 v1:172.21.15.137:6809/1833792554 exists,up 54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a'] 2024-09-16T02:47:41.900 INFO:tasks.osd_recovery:all are up! 2024-09-16T02:47:41.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 0 2024-09-16T02:47:42.054 INFO:teuthology.orchestra.run.smithi137.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T02:47:42.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 debug kick_recovery_wq 0 2024-09-16T02:47:42.222 INFO:teuthology.orchestra.run.smithi137.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T02:47:42.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 0 2024-09-16T02:47:42.390 INFO:teuthology.orchestra.run.smithi137.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T02:47:42.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 0 2024-09-16T02:47:42.556 INFO:teuthology.orchestra.run.smithi137.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-16T02:47:42.574 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2024-09-16T02:47:42.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:42.924 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:42.924 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:42.944 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2024-09-16T02:47:41.662574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":138788,"kb_used_data":13952,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188604892,"statfs":{"total":193273528320,"available":193131409408,"internally_reserved":0,"allocated":14286848,"data_stored":158042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2921,"num_objects_misplaced":1567,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":677,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002438"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6353,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:38.798881+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:38.798881+0000","last_peered":"2024-09-16T02:47:38.798881+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:38.798881+0000","last_undegraded":"2024-09-16T02:47:38.798881+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4488,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":794,"num_bytes_recovered":794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1666},{"shards":"2,3","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275939328,"data_stored":565136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640485,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78048,"kb_used_data":6604,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293792,"statfs":{"total":96636764160,"available":96556843008,"internally_reserved":0,"allocated":6762496,"data_stored":77397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673190,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60740,"kb_used_data":7348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311100,"statfs":{"total":96636764160,"available":96574566400,"internally_reserved":0,"allocated":7524352,"data_stored":80645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986178,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7405568,"data_stored":1808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6561792,"data_stored":1602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:42.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:43.278 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:43.297 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2024-09-16T02:47:41.662574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":138788,"kb_used_data":13952,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188604892,"statfs":{"total":193273528320,"available":193131409408,"internally_reserved":0,"allocated":14286848,"data_stored":158042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2921,"num_objects_misplaced":1567,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":677,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002438"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6353,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:38.798881+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:38.798881+0000","last_peered":"2024-09-16T02:47:38.798881+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:38.798881+0000","last_undegraded":"2024-09-16T02:47:38.798881+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4488,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":794,"num_bytes_recovered":794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1666},{"shards":"2,3","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275939328,"data_stored":565136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640485,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78048,"kb_used_data":6604,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293792,"statfs":{"total":96636764160,"available":96556843008,"internally_reserved":0,"allocated":6762496,"data_stored":77397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673190,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60740,"kb_used_data":7348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311100,"statfs":{"total":96636764160,"available":96574566400,"internally_reserved":0,"allocated":7524352,"data_stored":80645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986178,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7405568,"data_stored":1808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6561792,"data_stored":1602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:43.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:43.616 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:43.617 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:43.632 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2024-09-16T02:47:41.662574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":138788,"kb_used_data":13952,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188604892,"statfs":{"total":193273528320,"available":193131409408,"internally_reserved":0,"allocated":14286848,"data_stored":158042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2921,"num_objects_misplaced":1567,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":677,"num_bytes_recovered":677,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002438"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6353,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:38.798881+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:38.798881+0000","last_peered":"2024-09-16T02:47:38.798881+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:38.798881+0000","last_undegraded":"2024-09-16T02:47:38.798881+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4488,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":794,"num_bytes_recovered":794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1666},{"shards":"2,3","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275939328,"data_stored":565136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640485,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78048,"kb_used_data":6604,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94293792,"statfs":{"total":96636764160,"available":96556843008,"internally_reserved":0,"allocated":6762496,"data_stored":77397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673190,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60740,"kb_used_data":7348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311100,"statfs":{"total":96636764160,"available":96574566400,"internally_reserved":0,"allocated":7524352,"data_stored":80645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986178,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7405568,"data_stored":1808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6561792,"data_stored":1602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:43.632 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706670+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:47:37.706670+0000', 'last_peered': '2024-09-16T02:47:37.706670+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:47:37.706670+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 4044, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.709564+0000', 'last_change': '2024-09-16T02:47:26.507105+0000', 'last_active': '2024-09-16T02:47:37.709564+0000', 'last_peered': '2024-09-16T02:47:37.709564+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:47:37.709564+0000', 'last_undegraded': '2024-09-16T02:47:26.487051+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3985, 'num_objects_misplaced': 3985, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3985}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4116, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706453+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:47:37.706453+0000', 'last_peered': '2024-09-16T02:47:37.706453+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:47:37.706453+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4016, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.707306+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:47:37.707306+0000', 'last_peered': '2024-09-16T02:47:37.707306+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:47:37.707306+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:47:43.633 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4073, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706352+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:47:37.706352+0000', 'last_peered': '2024-09-16T02:47:37.706352+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:47:37.706352+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.634 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:47:43.634 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706337+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:47:37.706337+0000', 'last_peered': '2024-09-16T02:47:37.706337+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:47:37.706337+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.634 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-16T02:47:43.634 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "19'3910", 'reported_seq': 6353, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:47:38.798881+0000', 'last_change': '2024-09-16T02:47:37.847429+0000', 'last_active': '2024-09-16T02:47:38.798881+0000', 'last_peered': '2024-09-16T02:47:38.798881+0000', 'last_clean': '2024-09-16T02:46:35.256073+0000', 'last_became_active': '2024-09-16T02:47:26.478173+0000', 'last_became_peered': '2024-09-16T02:47:26.478173+0000', 'last_unstale': '2024-09-16T02:47:38.798881+0000', 'last_undegraded': '2024-09-16T02:47:38.798881+0000', 'last_fullsized': '2024-09-16T02:47:26.333712+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3910, 'log_dups_size': 0, 'ondisk_log_size': 3910, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001711783, 'stat_sum': {'num_bytes': 17010, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4488, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3910, 'num_write_kb': 3922, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 794, 'num_bytes_recovered': 794, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1666}, {'shards': '2,3', 'objects': 2244}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.634 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:47:43.634 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706307+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:47:37.706307+0000', 'last_peered': '2024-09-16T02:47:37.706307+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:47:37.706307+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:43.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:47:44.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:44.046 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28013/63846 objects degraded (43.876%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1123,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":143409152,"bytes_avail":193130119168,"bytes_total":193273528320,"degraded_objects":28013,"degraded_total":63846,"degraded_ratio":0.43875888857563511,"misplaced_objects":32501,"misplaced_total":63846,"misplaced_ratio":0.50905303386273215,"recovering_objects_per_sec":42,"recovering_bytes_per_sec":42,"recovering_keys_per_sec":0,"num_objects_recovered":343,"num_bytes_recovered":343,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:47:44.067 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:47:44.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:44.403 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:44.404 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:44.423 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2024-09-16T02:47:43.662962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":140048,"kb_used_data":15216,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188603632,"statfs":{"total":193273528320,"available":193130119168,"internally_reserved":0,"allocated":15581184,"data_stored":161420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2587,"num_objects_misplaced":1901,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002399"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":6353,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:38.798881+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:38.798881+0000","last_peered":"2024-09-16T02:47:38.798881+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:38.798881+0000","last_undegraded":"2024-09-16T02:47:38.798881+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4488,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":794,"num_bytes_recovered":794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1666},{"shards":"2,3","objects":2244}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":32501,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":813,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277307392,"data_stored":565470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640486,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79308,"kb_used_data":7868,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94292532,"statfs":{"total":96636764160,"available":96555552768,"internally_reserved":0,"allocated":8056832,"data_stored":80775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673190,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60740,"kb_used_data":7348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94311100,"statfs":{"total":96636764160,"available":96574566400,"internally_reserved":0,"allocated":7524352,"data_stored":80645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986178,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7405568,"data_stored":1808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7929856,"data_stored":1936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:47.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:47.790 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:47.790 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:47.809 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-09-16T02:47:47.664350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":31739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1177,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":147520,"kb_used_data":18528,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188596160,"statfs":{"total":193273528320,"available":193122467840,"internally_reserved":0,"allocated":18972672,"data_stored":162248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2587,"num_objects_misplaced":1139,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":724,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003050"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":7496,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:43.808514+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:43.808514+0000","last_peered":"2024-09-16T02:47:43.808514+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:43.808514+0000","last_undegraded":"2024-09-16T02:47:43.808514+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3726,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":1175,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2047},{"shards":"2,3","objects":1863}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":31739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1177,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280674304,"data_stored":566292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85060,"kb_used_data":9460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286780,"statfs":{"total":96636764160,"available":96549662720,"internally_reserved":0,"allocated":9687040,"data_stored":81173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673191,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62460,"kb_used_data":9068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309380,"statfs":{"total":96636764160,"available":96572805120,"internally_reserved":0,"allocated":9285632,"data_stored":81075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986179,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9195520,"data_stored":2245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9506816,"data_stored":2321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:47.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:48.142 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:48.165 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-09-16T02:47:47.664350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":31739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1177,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":147520,"kb_used_data":18528,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188596160,"statfs":{"total":193273528320,"available":193122467840,"internally_reserved":0,"allocated":18972672,"data_stored":162248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2587,"num_objects_misplaced":1139,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":724,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003050"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":7496,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:43.808514+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:43.808514+0000","last_peered":"2024-09-16T02:47:43.808514+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:43.808514+0000","last_undegraded":"2024-09-16T02:47:43.808514+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3726,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":1175,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2047},{"shards":"2,3","objects":1863}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":31739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1177,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280674304,"data_stored":566292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85060,"kb_used_data":9460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286780,"statfs":{"total":96636764160,"available":96549662720,"internally_reserved":0,"allocated":9687040,"data_stored":81173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673191,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62460,"kb_used_data":9068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309380,"statfs":{"total":96636764160,"available":96572805120,"internally_reserved":0,"allocated":9285632,"data_stored":81075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986179,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9195520,"data_stored":2245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9506816,"data_stored":2321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:48.165 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:47:48.165 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706670+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:47:37.706670+0000', 'last_peered': '2024-09-16T02:47:37.706670+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:47:37.706670+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 4044, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.709564+0000', 'last_change': '2024-09-16T02:47:26.507105+0000', 'last_active': '2024-09-16T02:47:37.709564+0000', 'last_peered': '2024-09-16T02:47:37.709564+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:47:37.709564+0000', 'last_undegraded': '2024-09-16T02:47:26.487051+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3985, 'num_objects_misplaced': 3985, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3985}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4116, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706453+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:47:37.706453+0000', 'last_peered': '2024-09-16T02:47:37.706453+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:47:37.706453+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4016, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.707306+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:47:37.707306+0000', 'last_peered': '2024-09-16T02:47:37.707306+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:47:37.707306+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4073, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706352+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:47:37.706352+0000', 'last_peered': '2024-09-16T02:47:37.706352+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:47:37.706352+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706337+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:47:37.706337+0000', 'last_peered': '2024-09-16T02:47:37.706337+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:47:37.706337+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-16T02:47:48.167 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "19'3910", 'reported_seq': 7496, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:47:43.808514+0000', 'last_change': '2024-09-16T02:47:37.847429+0000', 'last_active': '2024-09-16T02:47:43.808514+0000', 'last_peered': '2024-09-16T02:47:43.808514+0000', 'last_clean': '2024-09-16T02:46:35.256073+0000', 'last_became_active': '2024-09-16T02:47:26.478173+0000', 'last_became_peered': '2024-09-16T02:47:26.478173+0000', 'last_unstale': '2024-09-16T02:47:43.808514+0000', 'last_undegraded': '2024-09-16T02:47:43.808514+0000', 'last_fullsized': '2024-09-16T02:47:26.333712+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3910, 'log_dups_size': 0, 'ondisk_log_size': 3910, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001711783, 'stat_sum': {'num_bytes': 17010, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3726, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3910, 'num_write_kb': 3922, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1175, 'num_bytes_recovered': 1175, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2047}, {'shards': '2,3', 'objects': 1863}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.167 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:47:48.167 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706307+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:47:37.706307+0000', 'last_peered': '2024-09-16T02:47:37.706307+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:47:37.706307+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:48.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:47:48.586 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:48.586 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28013/63846 objects degraded (43.876%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1128,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":151060480,"bytes_avail":193122467840,"bytes_total":193273528320,"degraded_objects":28013,"degraded_total":63846,"degraded_ratio":0.43875888857563511,"misplaced_objects":31739,"misplaced_total":63846,"misplaced_ratio":0.49711806534473579,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":72,"recovering_keys_per_sec":0,"num_objects_recovered":724,"num_bytes_recovered":724,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:47:48.607 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:47:48.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:48.948 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:48.948 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:48.969 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-09-16T02:47:47.664350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":31739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1177,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":147520,"kb_used_data":18528,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188596160,"statfs":{"total":193273528320,"available":193122467840,"internally_reserved":0,"allocated":18972672,"data_stored":162248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2587,"num_objects_misplaced":1139,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":724,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003050"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":7496,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:43.808514+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:43.808514+0000","last_peered":"2024-09-16T02:47:43.808514+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:43.808514+0000","last_undegraded":"2024-09-16T02:47:43.808514+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3726,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":1175,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2047},{"shards":"2,3","objects":1863}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":31739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1177,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280674304,"data_stored":566292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85060,"kb_used_data":9460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286780,"statfs":{"total":96636764160,"available":96549662720,"internally_reserved":0,"allocated":9687040,"data_stored":81173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673191,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62460,"kb_used_data":9068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94309380,"statfs":{"total":96636764160,"available":96572805120,"internally_reserved":0,"allocated":9285632,"data_stored":81075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986179,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9195520,"data_stored":2245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9506816,"data_stored":2321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:51.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:52.322 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:52.322 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:52.342 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-09-16T02:47:51.665555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":30885,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":1621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":149272,"kb_used_data":20280,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188594408,"statfs":{"total":193273528320,"available":193120673792,"internally_reserved":0,"allocated":20766720,"data_stored":162686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":8777,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:48.815645+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:48.815645+0000","last_peered":"2024-09-16T02:47:48.815645+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:48.815645+0000","last_undegraded":"2024-09-16T02:47:48.815645+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2872,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1602,"num_bytes_recovered":1602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2474},{"shards":"2,3","objects":1436}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":30885,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":1621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282505216,"data_stored":566739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85060,"kb_used_data":9460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286780,"statfs":{"total":96636764160,"available":96549662720,"internally_reserved":0,"allocated":9687040,"data_stored":81173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":11079680,"data_stored":81513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986180,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11026432,"data_stored":2692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9506816,"data_stored":2321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:52.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:52.677 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:52.677 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:52.699 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-09-16T02:47:51.665555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":30885,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":1621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":149272,"kb_used_data":20280,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188594408,"statfs":{"total":193273528320,"available":193120673792,"internally_reserved":0,"allocated":20766720,"data_stored":162686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":8777,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:48.815645+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:48.815645+0000","last_peered":"2024-09-16T02:47:48.815645+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:48.815645+0000","last_undegraded":"2024-09-16T02:47:48.815645+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2872,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1602,"num_bytes_recovered":1602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2474},{"shards":"2,3","objects":1436}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":30885,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":1621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282505216,"data_stored":566739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85060,"kb_used_data":9460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286780,"statfs":{"total":96636764160,"available":96549662720,"internally_reserved":0,"allocated":9687040,"data_stored":81173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":11079680,"data_stored":81513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986180,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11026432,"data_stored":2692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9506816,"data_stored":2321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:52.700 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:47:52.700 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706670+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:47:37.706670+0000', 'last_peered': '2024-09-16T02:47:37.706670+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:47:37.706670+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.700 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:47:52.700 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 4044, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.709564+0000', 'last_change': '2024-09-16T02:47:26.507105+0000', 'last_active': '2024-09-16T02:47:37.709564+0000', 'last_peered': '2024-09-16T02:47:37.709564+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:47:37.709564+0000', 'last_undegraded': '2024-09-16T02:47:26.487051+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3985, 'num_objects_misplaced': 3985, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3985}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.700 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:47:52.700 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4116, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706453+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:47:37.706453+0000', 'last_peered': '2024-09-16T02:47:37.706453+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:47:37.706453+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4016, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.707306+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:47:37.707306+0000', 'last_peered': '2024-09-16T02:47:37.707306+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:47:37.707306+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4073, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706352+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:47:37.706352+0000', 'last_peered': '2024-09-16T02:47:37.706352+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:47:37.706352+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706337+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:47:37.706337+0000', 'last_peered': '2024-09-16T02:47:37.706337+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:47:37.706337+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "19'3910", 'reported_seq': 8777, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:47:48.815645+0000', 'last_change': '2024-09-16T02:47:37.847429+0000', 'last_active': '2024-09-16T02:47:48.815645+0000', 'last_peered': '2024-09-16T02:47:48.815645+0000', 'last_clean': '2024-09-16T02:46:35.256073+0000', 'last_became_active': '2024-09-16T02:47:26.478173+0000', 'last_became_peered': '2024-09-16T02:47:26.478173+0000', 'last_unstale': '2024-09-16T02:47:48.815645+0000', 'last_undegraded': '2024-09-16T02:47:48.815645+0000', 'last_fullsized': '2024-09-16T02:47:26.333712+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3910, 'log_dups_size': 0, 'ondisk_log_size': 3910, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001711783, 'stat_sum': {'num_bytes': 17010, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2872, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3910, 'num_write_kb': 3922, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1602, 'num_bytes_recovered': 1602, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2474}, {'shards': '2,3', 'objects': 1436}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:47:52.701 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706307+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:47:37.706307+0000', 'last_peered': '2024-09-16T02:47:37.706307+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:47:37.706307+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:52.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:47:53.119 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:53.119 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28013/63846 objects degraded (43.876%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1132,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":152854528,"bytes_avail":193120673792,"bytes_total":193273528320,"degraded_objects":28013,"degraded_total":63846,"degraded_ratio":0.43875888857563511,"misplaced_objects":30885,"misplaced_total":63846,"misplaced_ratio":0.48374212949910722,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":67,"recovering_keys_per_sec":0,"num_objects_recovered":808,"num_bytes_recovered":808,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:47:53.140 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:47:53.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:53.467 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:53.468 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:53.486 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-09-16T02:47:51.665555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":30885,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":1621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":149272,"kb_used_data":20280,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188594408,"statfs":{"total":193273528320,"available":193120673792,"internally_reserved":0,"allocated":20766720,"data_stored":162686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003584"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":8777,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:48.815645+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:48.815645+0000","last_peered":"2024-09-16T02:47:48.815645+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:48.815645+0000","last_undegraded":"2024-09-16T02:47:48.815645+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2872,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1602,"num_bytes_recovered":1602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2474},{"shards":"2,3","objects":1436}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":30885,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1604,"num_bytes_recovered":1621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282505216,"data_stored":566739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85060,"kb_used_data":9460,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286780,"statfs":{"total":96636764160,"available":96549662720,"internally_reserved":0,"allocated":9687040,"data_stored":81173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":11079680,"data_stored":81513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986180,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11026432,"data_stored":2692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9506816,"data_stored":2321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:56.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:56.840 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:56.860 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":591,"stamp":"2024-09-16T02:47:55.666510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":29813,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":151168,"kb_used_data":22176,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188592512,"statfs":{"total":193273528320,"available":193118732288,"internally_reserved":0,"allocated":22708224,"data_stored":163160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2688,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1344,"num_bytes_recovered":1344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003547"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":10385,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:53.813561+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:53.813561+0000","last_peered":"2024-09-16T02:47:53.813561+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:53.813561+0000","last_undegraded":"2024-09-16T02:47:53.813561+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1800,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":2138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3010},{"shards":"2,3","objects":900}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":29813,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":284512256,"data_stored":567229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640488,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86956,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94284884,"statfs":{"total":96636764160,"available":96547721216,"internally_reserved":0,"allocated":11628544,"data_stored":81647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":11079680,"data_stored":81513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986180,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11026432,"data_stored":2692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11513856,"data_stored":2811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:56.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:57.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:57.211 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:57.232 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":591,"stamp":"2024-09-16T02:47:55.666510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":29813,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":151168,"kb_used_data":22176,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188592512,"statfs":{"total":193273528320,"available":193118732288,"internally_reserved":0,"allocated":22708224,"data_stored":163160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2688,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1344,"num_bytes_recovered":1344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003547"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":10385,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:53.813561+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:53.813561+0000","last_peered":"2024-09-16T02:47:53.813561+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:53.813561+0000","last_undegraded":"2024-09-16T02:47:53.813561+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1800,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":2138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3010},{"shards":"2,3","objects":900}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":29813,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":284512256,"data_stored":567229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640488,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86956,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94284884,"statfs":{"total":96636764160,"available":96547721216,"internally_reserved":0,"allocated":11628544,"data_stored":81647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64212,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94307628,"statfs":{"total":96636764160,"available":96571011072,"internally_reserved":0,"allocated":11079680,"data_stored":81513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986180,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11026432,"data_stored":2692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11513856,"data_stored":2811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706670+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:47:37.706670+0000', 'last_peered': '2024-09-16T02:47:37.706670+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:47:37.706670+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 4044, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.709564+0000', 'last_change': '2024-09-16T02:47:26.507105+0000', 'last_active': '2024-09-16T02:47:37.709564+0000', 'last_peered': '2024-09-16T02:47:37.709564+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:47:37.709564+0000', 'last_undegraded': '2024-09-16T02:47:26.487051+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3985, 'num_objects_misplaced': 3985, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3985}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4116, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706453+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:47:37.706453+0000', 'last_peered': '2024-09-16T02:47:37.706453+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:47:37.706453+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:47:57.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4016, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.707306+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:47:37.707306+0000', 'last_peered': '2024-09-16T02:47:37.707306+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:47:37.707306+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4073, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706352+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:47:37.706352+0000', 'last_peered': '2024-09-16T02:47:37.706352+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:47:37.706352+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706337+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:47:37.706337+0000', 'last_peered': '2024-09-16T02:47:37.706337+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:47:37.706337+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "19'3910", 'reported_seq': 10385, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:47:53.813561+0000', 'last_change': '2024-09-16T02:47:37.847429+0000', 'last_active': '2024-09-16T02:47:53.813561+0000', 'last_peered': '2024-09-16T02:47:53.813561+0000', 'last_clean': '2024-09-16T02:46:35.256073+0000', 'last_became_active': '2024-09-16T02:47:26.478173+0000', 'last_became_peered': '2024-09-16T02:47:26.478173+0000', 'last_unstale': '2024-09-16T02:47:53.813561+0000', 'last_undegraded': '2024-09-16T02:47:53.813561+0000', 'last_fullsized': '2024-09-16T02:47:26.333712+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3910, 'log_dups_size': 0, 'ondisk_log_size': 3910, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001711783, 'stat_sum': {'num_bytes': 17010, 'num_objects': 3910, 'num_object_clones': 0, 'num_object_copies': 7820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1800, 'num_objects_unfound': 0, 'num_objects_dirty': 3910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3910, 'num_write_kb': 3922, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2138, 'num_bytes_recovered': 2138, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3010}, {'shards': '2,3', 'objects': 900}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:47:57.234 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706307+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:47:37.706307+0000', 'last_peered': '2024-09-16T02:47:37.706307+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:47:37.706307+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:47:57.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:47:57.666 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:57.666 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28013/63846 objects degraded (43.876%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1137,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":154796032,"bytes_avail":193118732288,"bytes_total":193273528320,"degraded_objects":28013,"degraded_total":63846,"degraded_ratio":0.43875888857563511,"misplaced_objects":29813,"misplaced_total":63846,"misplaced_ratio":0.4669517275945243,"recovering_objects_per_sec":111,"recovering_bytes_per_sec":111,"recovering_keys_per_sec":0,"num_objects_recovered":1344,"num_bytes_recovered":1344,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:47:57.687 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:47:57.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:47:58.025 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:47:58.026 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:47:58.045 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2024-09-16T02:47:57.667116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":29813,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":164372,"kb_used_data":27060,"kb_used_omap":3,"kb_used_meta":137276,"kb_avail":188579308,"statfs":{"total":193273528320,"available":193105211392,"internally_reserved":0,"allocated":27709440,"data_stored":164381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":140571530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1926,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003586"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":10385,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:53.813561+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:53.813561+0000","last_peered":"2024-09-16T02:47:53.813561+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:53.813561+0000","last_undegraded":"2024-09-16T02:47:53.813561+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1800,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2138,"num_bytes_recovered":2138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3010},{"shards":"2,3","objects":900}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":29813,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2140,"num_bytes_recovered":2157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289669120,"data_stored":568488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964116999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93772,"kb_used_data":14012,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94278068,"statfs":{"total":96636764160,"available":96540741632,"internally_reserved":0,"allocated":14348288,"data_stored":82311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673193,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70600,"kb_used_data":13048,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301240,"statfs":{"total":96636764160,"available":96564469760,"internally_reserved":0,"allocated":13361152,"data_stored":82070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986181,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13352960,"data_stored":3260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14344192,"data_stored":3502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:01.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:01.365 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:01.365 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:01.385 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":595,"stamp":"2024-09-16T02:48:00.788337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":28299,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2897,"num_bytes_recovered":2914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":164372,"kb_used_data":27060,"kb_used_omap":3,"kb_used_meta":137276,"kb_avail":188579308,"statfs":{"total":193273528320,"available":193105211392,"internally_reserved":0,"allocated":27709440,"data_stored":164381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":140571530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2586,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1293,"num_bytes_recovered":1293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.122781"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":12656,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:47:58.820621+0000","last_change":"2024-09-16T02:47:37.847429+0000","last_active":"2024-09-16T02:47:58.820621+0000","last_peered":"2024-09-16T02:47:58.820621+0000","last_clean":"2024-09-16T02:46:35.256073+0000","last_became_active":"2024-09-16T02:47:26.478173+0000","last_became_peered":"2024-09-16T02:47:26.478173+0000","last_unstale":"2024-09-16T02:47:58.820621+0000","last_undegraded":"2024-09-16T02:47:58.820621+0000","last_fullsized":"2024-09-16T02:47:26.333712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":286,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2895,"num_bytes_recovered":2895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3767},{"shards":"2,3","objects":143}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":28299,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2897,"num_bytes_recovered":2914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289669120,"data_stored":568488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93772,"kb_used_data":14012,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94278068,"statfs":{"total":96636764160,"available":96540741632,"internally_reserved":0,"allocated":14348288,"data_stored":82311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673193,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70600,"kb_used_data":13048,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94301240,"statfs":{"total":96636764160,"available":96564469760,"internally_reserved":0,"allocated":13361152,"data_stored":82070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986181,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13352960,"data_stored":3260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14344192,"data_stored":3502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:01.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:01.707 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:01.707 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:01.727 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-09-16T02:48:01.668103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":28013,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":16157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":154972,"kb_used_data":30140,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188588708,"statfs":{"total":193273528320,"available":193114836992,"internally_reserved":0,"allocated":30863360,"data_stored":183883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2872,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":14536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001989"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":28013,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":16157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":288784384,"data_stored":581369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93772,"kb_used_data":14012,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94278068,"statfs":{"total":96636764160,"available":96540741632,"internally_reserved":0,"allocated":14348288,"data_stored":82311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61200,"kb_used_data":16128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94310640,"statfs":{"total":96636764160,"available":96574095360,"internally_reserved":0,"allocated":16515072,"data_stored":101572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986182,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16396288,"data_stored":17100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14344192,"data_stored":3502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127057920,"data_stored":279904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:01.727 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706670+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:47:37.706670+0000', 'last_peered': '2024-09-16T02:47:37.706670+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:47:37.706670+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 4044, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.709564+0000', 'last_change': '2024-09-16T02:47:26.507105+0000', 'last_active': '2024-09-16T02:47:37.709564+0000', 'last_peered': '2024-09-16T02:47:37.709564+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:47:37.709564+0000', 'last_undegraded': '2024-09-16T02:47:26.487051+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3985, 'num_objects_misplaced': 3985, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3985}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4116, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706453+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:47:37.706453+0000', 'last_peered': '2024-09-16T02:47:37.706453+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:47:37.706453+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4016, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.707306+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:47:37.707306+0000', 'last_peered': '2024-09-16T02:47:37.707306+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:47:37.707306+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4073, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706352+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:47:37.706352+0000', 'last_peered': '2024-09-16T02:47:37.706352+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:47:37.706352+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:01.728 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:01.729 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706337+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:47:37.706337+0000', 'last_peered': '2024-09-16T02:47:37.706337+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:47:37.706337+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:01.729 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:01.729 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:47:37.706307+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:47:37.706307+0000', 'last_peered': '2024-09-16T02:47:37.706307+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:47:37.706307+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:01.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:02.137 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:02.137 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28013/63846 objects degraded (43.876%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1141,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":158691328,"bytes_avail":193114836992,"bytes_total":193273528320,"degraded_objects":28013,"degraded_total":63846,"degraded_ratio":0.43875888857563511,"misplaced_objects":28013,"misplaced_total":63846,"misplaced_ratio":0.43875888857563511,"recovering_objects_per_sec":179,"recovering_bytes_per_sec":1816,"recovering_keys_per_sec":0,"num_objects_recovered":1436,"num_bytes_recovered":14536,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:02.158 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:02.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:02.481 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:02.500 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-09-16T02:48:01.668103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":28013,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":16157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":154972,"kb_used_data":30140,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188588708,"statfs":{"total":193273528320,"available":193114836992,"internally_reserved":0,"allocated":30863360,"data_stored":183883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2872,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":14536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001989"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706670+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:47:37.706670+0000","last_peered":"2024-09-16T02:47:37.706670+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:47:37.706670+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.709564+0000","last_change":"2024-09-16T02:47:26.507105+0000","last_active":"2024-09-16T02:47:37.709564+0000","last_peered":"2024-09-16T02:47:37.709564+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:47:37.709564+0000","last_undegraded":"2024-09-16T02:47:26.487051+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3985,"num_objects_misplaced":3985,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3985}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706453+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:47:37.706453+0000","last_peered":"2024-09-16T02:47:37.706453+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:47:37.706453+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4016,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.707306+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:47:37.707306+0000","last_peered":"2024-09-16T02:47:37.707306+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:47:37.707306+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4073,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706352+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:47:37.706352+0000","last_peered":"2024-09-16T02:47:37.706352+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:47:37.706352+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706337+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:47:37.706337+0000","last_peered":"2024-09-16T02:47:37.706337+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:47:37.706337+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:47:37.706307+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:47:37.706307+0000","last_peered":"2024-09-16T02:47:37.706307+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:47:37.706307+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28013,"num_objects_misplaced":28013,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3040,"num_bytes_recovered":16157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":288784384,"data_stored":581369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93772,"kb_used_data":14012,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94278068,"statfs":{"total":96636764160,"available":96540741632,"internally_reserved":0,"allocated":14348288,"data_stored":82311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61200,"kb_used_data":16128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94310640,"statfs":{"total":96636764160,"available":96574095360,"internally_reserved":0,"allocated":16515072,"data_stored":101572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986182,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16396288,"data_stored":17100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14344192,"data_stored":3502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127057920,"data_stored":279904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:05.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:05.861 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:05.861 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:05.883 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-09-16T02:48:05.669330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31582,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3248,"num_bytes_recovered":16365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":140832,"kb_used_data":32640,"kb_used_omap":3,"kb_used_meta":108156,"kb_avail":188602848,"statfs":{"total":193273528320,"available":193129316352,"internally_reserved":0,"allocated":33423360,"data_stored":203240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":110752650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":1769,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1108,"num_bytes_recovered":14208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002213"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4673,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:03.805391+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:03.805391+0000","last_peered":"2024-09-16T02:48:03.805391+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:03.805391+0000","last_undegraded":"2024-09-16T02:48:03.805391+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7554,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":208},{"shards":"2,3","objects":3777}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31582,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3248,"num_bytes_recovered":16365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":291143680,"data_stored":595042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640490,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79632,"kb_used_data":16512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94292208,"statfs":{"total":96636764160,"available":96555220992,"internally_reserved":0,"allocated":16908288,"data_stored":101668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61200,"kb_used_data":16128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94310640,"statfs":{"total":96636764160,"available":96574095360,"internally_reserved":0,"allocated":16515072,"data_stored":101572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986182,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16396288,"data_stored":17100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16703488,"data_stored":17175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127057920,"data_stored":279904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:05.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:06.233 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:06.233 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:06.254 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-09-16T02:48:05.669330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31582,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3248,"num_bytes_recovered":16365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":140832,"kb_used_data":32640,"kb_used_omap":3,"kb_used_meta":108156,"kb_avail":188602848,"statfs":{"total":193273528320,"available":193129316352,"internally_reserved":0,"allocated":33423360,"data_stored":203240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":110752650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":1769,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1108,"num_bytes_recovered":14208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002213"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4673,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:03.805391+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:03.805391+0000","last_peered":"2024-09-16T02:48:03.805391+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:03.805391+0000","last_undegraded":"2024-09-16T02:48:03.805391+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7554,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":208},{"shards":"2,3","objects":3777}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31582,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3248,"num_bytes_recovered":16365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":291143680,"data_stored":595042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640490,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79632,"kb_used_data":16512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94292208,"statfs":{"total":96636764160,"available":96555220992,"internally_reserved":0,"allocated":16908288,"data_stored":101668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61200,"kb_used_data":16128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94310640,"statfs":{"total":96636764160,"available":96574095360,"internally_reserved":0,"allocated":16515072,"data_stored":101572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986182,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16396288,"data_stored":17100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16703488,"data_stored":17175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127057920,"data_stored":279904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:06.254 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:06.254 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 4673, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:03.805391+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:03.805391+0000', 'last_peered': '2024-09-16T02:48:03.805391+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:03.805391+0000', 'last_undegraded': '2024-09-16T02:48:03.805391+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7554, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 208, 'num_bytes_recovered': 208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 208}, {'shards': '2,3', 'objects': 3777}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:06.255 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:06.256 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:06.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:06.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:06.654 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded","count":6},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1146,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":144211968,"bytes_avail":193129316352,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":31582,"misplaced_total":63846,"misplaced_ratio":0.49465902327475481,"recovering_objects_per_sec":138,"recovering_bytes_per_sec":1775,"recovering_keys_per_sec":0,"num_objects_recovered":1108,"num_bytes_recovered":14208,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:06.674 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:06.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:06.996 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:07.016 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-09-16T02:48:05.669330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31582,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3248,"num_bytes_recovered":16365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":140832,"kb_used_data":32640,"kb_used_omap":3,"kb_used_meta":108156,"kb_avail":188602848,"statfs":{"total":193273528320,"available":193129316352,"internally_reserved":0,"allocated":33423360,"data_stored":203240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":110752650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":1769,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1108,"num_bytes_recovered":14208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002213"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4673,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:03.805391+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:03.805391+0000","last_peered":"2024-09-16T02:48:03.805391+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:03.805391+0000","last_undegraded":"2024-09-16T02:48:03.805391+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7554,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":208},{"shards":"2,3","objects":3777}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31582,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3248,"num_bytes_recovered":16365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":291143680,"data_stored":595042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640490,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79632,"kb_used_data":16512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94292208,"statfs":{"total":96636764160,"available":96555220992,"internally_reserved":0,"allocated":16908288,"data_stored":101668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673194,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61200,"kb_used_data":16128,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94310640,"statfs":{"total":96636764160,"available":96574095360,"internally_reserved":0,"allocated":16515072,"data_stored":101572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986182,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16396288,"data_stored":17100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16703488,"data_stored":17175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127057920,"data_stored":279904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:10.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:10.360 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:10.361 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:10.380 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2024-09-16T02:48:09.670338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":146980,"kb_used_data":34628,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188596700,"statfs":{"total":193273528320,"available":193123020800,"internally_reserved":0,"allocated":35459072,"data_stored":203737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":2801,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":13692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.888495"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":5396,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:08.827489+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:08.827489+0000","last_peered":"2024-09-16T02:48:08.827489+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:08.827489+0000","last_undegraded":"2024-09-16T02:48:08.827489+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7072,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":449},{"shards":"2,3","objects":3536}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281042944,"data_stored":579479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84764,"kb_used_data":17484,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94287076,"statfs":{"total":96636764160,"available":96549965824,"internally_reserved":0,"allocated":17903616,"data_stored":101911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673195,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62216,"kb_used_data":17144,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94309624,"statfs":{"total":96636764160,"available":96573054976,"internally_reserved":0,"allocated":17555456,"data_stored":101826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986183,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17416192,"data_stored":17349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17682432,"data_stored":17414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:10.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:10.706 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:10.706 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:10.725 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2024-09-16T02:48:09.670338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":146980,"kb_used_data":34628,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188596700,"statfs":{"total":193273528320,"available":193123020800,"internally_reserved":0,"allocated":35459072,"data_stored":203737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":2801,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":13692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.888495"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":5396,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:08.827489+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:08.827489+0000","last_peered":"2024-09-16T02:48:08.827489+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:08.827489+0000","last_undegraded":"2024-09-16T02:48:08.827489+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7072,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":449},{"shards":"2,3","objects":3536}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281042944,"data_stored":579479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84764,"kb_used_data":17484,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94287076,"statfs":{"total":96636764160,"available":96549965824,"internally_reserved":0,"allocated":17903616,"data_stored":101911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673195,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62216,"kb_used_data":17144,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94309624,"statfs":{"total":96636764160,"available":96573054976,"internally_reserved":0,"allocated":17555456,"data_stored":101826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986183,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17416192,"data_stored":17349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17682432,"data_stored":17414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:10.726 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:10.726 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:10.726 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 5396, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:08.827489+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:08.827489+0000', 'last_peered': '2024-09-16T02:48:08.827489+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:08.827489+0000', 'last_undegraded': '2024-09-16T02:48:08.827489+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7072, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 449, 'num_bytes_recovered': 449, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 449}, {'shards': '2,3', 'objects': 3536}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:10.727 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:10.728 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:10.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:11.182 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:11.182 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded","count":6},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1150,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":150507520,"bytes_avail":193123020800,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":31100,"misplaced_total":63846,"misplaced_ratio":0.48710960749303012,"recovering_objects_per_sec":59,"recovering_bytes_per_sec":1384,"recovering_keys_per_sec":0,"num_objects_recovered":592,"num_bytes_recovered":13692,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:11.205 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:11.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:11.520 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:11.521 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:11.541 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2024-09-16T02:48:09.670338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":146980,"kb_used_data":34628,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188596700,"statfs":{"total":193273528320,"available":193123020800,"internally_reserved":0,"allocated":35459072,"data_stored":203737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":2801,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":13692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.888495"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":5396,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:08.827489+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:08.827489+0000","last_peered":"2024-09-16T02:48:08.827489+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:08.827489+0000","last_undegraded":"2024-09-16T02:48:08.827489+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7072,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":449},{"shards":"2,3","objects":3536}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281042944,"data_stored":579479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640491,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84764,"kb_used_data":17484,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94287076,"statfs":{"total":96636764160,"available":96549965824,"internally_reserved":0,"allocated":17903616,"data_stored":101911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673195,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62216,"kb_used_data":17144,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94309624,"statfs":{"total":96636764160,"available":96573054976,"internally_reserved":0,"allocated":17555456,"data_stored":101826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986183,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17416192,"data_stored":17349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17682432,"data_stored":17414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:14.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:14.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:14.880 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:14.899 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2024-09-16T02:48:13.671570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":149004,"kb_used_data":36652,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188594676,"statfs":{"total":193273528320,"available":193120948224,"internally_reserved":0,"allocated":37531648,"data_stored":204243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":3087,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":5396,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:08.827489+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:08.827489+0000","last_peered":"2024-09-16T02:48:08.827489+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:08.827489+0000","last_undegraded":"2024-09-16T02:48:08.827489+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7072,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":449},{"shards":"2,3","objects":3536}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":283095040,"data_stored":579980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640492,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85740,"kb_used_data":18460,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94286100,"statfs":{"total":96636764160,"available":96548966400,"internally_reserved":0,"allocated":18903040,"data_stored":102155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673196,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63264,"kb_used_data":18192,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94308576,"statfs":{"total":96636764160,"available":96571981824,"internally_reserved":0,"allocated":18628608,"data_stored":102088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986184,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18481152,"data_stored":17609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18669568,"data_stored":17655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:14.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:15.221 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:15.221 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:15.241 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2024-09-16T02:48:13.671570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":149004,"kb_used_data":36652,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188594676,"statfs":{"total":193273528320,"available":193120948224,"internally_reserved":0,"allocated":37531648,"data_stored":204243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":3087,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":5396,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:08.827489+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:08.827489+0000","last_peered":"2024-09-16T02:48:08.827489+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:08.827489+0000","last_undegraded":"2024-09-16T02:48:08.827489+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7072,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":449},{"shards":"2,3","objects":3536}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":31100,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3489,"num_bytes_recovered":16606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":283095040,"data_stored":579980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640492,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85740,"kb_used_data":18460,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94286100,"statfs":{"total":96636764160,"available":96548966400,"internally_reserved":0,"allocated":18903040,"data_stored":102155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673196,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63264,"kb_used_data":18192,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94308576,"statfs":{"total":96636764160,"available":96571981824,"internally_reserved":0,"allocated":18628608,"data_stored":102088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986184,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18481152,"data_stored":17609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18669568,"data_stored":17655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:15.242 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:15.242 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:15.242 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:15.242 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 5396, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:08.827489+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:08.827489+0000', 'last_peered': '2024-09-16T02:48:08.827489+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:08.827489+0000', 'last_undegraded': '2024-09-16T02:48:08.827489+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7072, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 449, 'num_bytes_recovered': 449, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 449}, {'shards': '2,3', 'objects': 3536}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:15.242 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:15.243 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:15.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:15.700 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:15.700 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded","count":6},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1155,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":152580096,"bytes_avail":193120948224,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":31100,"misplaced_total":63846,"misplaced_ratio":0.48710960749303012,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":37,"recovering_keys_per_sec":0,"num_objects_recovered":449,"num_bytes_recovered":449,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:15.720 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:15.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:16.036 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:16.036 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:16.056 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":603,"stamp":"2024-09-16T02:48:15.672178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30610,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3734,"num_bytes_recovered":16851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":149004,"kb_used_data":36652,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188594676,"statfs":{"total":193273528320,"available":193120948224,"internally_reserved":0,"allocated":37531648,"data_stored":204243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":2597,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":694,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003652"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":6131,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:13.832159+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:13.832159+0000","last_peered":"2024-09-16T02:48:13.832159+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:13.832159+0000","last_undegraded":"2024-09-16T02:48:13.832159+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6582,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":694,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":694},{"shards":"2,3","objects":3291}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30610,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3734,"num_bytes_recovered":16851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":283095040,"data_stored":579980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640492,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85740,"kb_used_data":18460,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94286100,"statfs":{"total":96636764160,"available":96548966400,"internally_reserved":0,"allocated":18903040,"data_stored":102155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673196,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63264,"kb_used_data":18192,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94308576,"statfs":{"total":96636764160,"available":96571981824,"internally_reserved":0,"allocated":18628608,"data_stored":102088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986184,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18481152,"data_stored":17609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18669568,"data_stored":17655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:19.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:19.410 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:19.410 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:19.430 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2024-09-16T02:48:17.672778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30610,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3734,"num_bytes_recovered":16851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":155308,"kb_used_data":38796,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188588372,"statfs":{"total":193273528320,"available":193114492928,"internally_reserved":0,"allocated":39727104,"data_stored":204779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-972,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":486,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003448"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":6131,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:13.832159+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:13.832159+0000","last_peered":"2024-09-16T02:48:13.832159+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:13.832159+0000","last_undegraded":"2024-09-16T02:48:13.832159+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6582,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":694,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":694},{"shards":"2,3","objects":3291}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30610,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3734,"num_bytes_recovered":16851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285298688,"data_stored":580518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640493,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86796,"kb_used_data":19516,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94285044,"statfs":{"total":96636764160,"available":96547885056,"internally_reserved":0,"allocated":19984384,"data_stored":102419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673197,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68512,"kb_used_data":19280,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94303328,"statfs":{"total":96636764160,"available":96566607872,"internally_reserved":0,"allocated":19742720,"data_stored":102360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986185,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19587072,"data_stored":17879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19767296,"data_stored":17923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:19.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:19.771 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:19.771 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:19.791 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-09-16T02:48:19.673451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":155308,"kb_used_data":38796,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188588372,"statfs":{"total":193273528320,"available":193114492928,"internally_reserved":0,"allocated":39727104,"data_stored":204779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1512,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003721"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":6941,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:18.850547+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:18.850547+0000","last_peered":"2024-09-16T02:48:18.850547+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:18.850547+0000","last_undegraded":"2024-09-16T02:48:18.850547+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6042,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":964},{"shards":"2,3","objects":3021}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285298688,"data_stored":580518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640493,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86796,"kb_used_data":19516,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94285044,"statfs":{"total":96636764160,"available":96547885056,"internally_reserved":0,"allocated":19984384,"data_stored":102419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673197,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68512,"kb_used_data":19280,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94303328,"statfs":{"total":96636764160,"available":96566607872,"internally_reserved":0,"allocated":19742720,"data_stored":102360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986185,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19587072,"data_stored":17879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19767296,"data_stored":17923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:19.792 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:19.792 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:19.792 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:19.792 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 6941, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:18.850547+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:18.850547+0000', 'last_peered': '2024-09-16T02:48:18.850547+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:18.850547+0000', 'last_undegraded': '2024-09-16T02:48:18.850547+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6042, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 964, 'num_bytes_recovered': 964, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 964}, {'shards': '2,3', 'objects': 3021}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:19.793 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:19.794 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:19.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:20.183 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:20.183 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded","count":6},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1159,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":159035392,"bytes_avail":193114492928,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":30070,"misplaced_total":63846,"misplaced_ratio":0.47097703849888795,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":62,"recovering_keys_per_sec":0,"num_objects_recovered":756,"num_bytes_recovered":756,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:20.203 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:20.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:20.561 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-09-16T02:48:19.673451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":155308,"kb_used_data":38796,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188588372,"statfs":{"total":193273528320,"available":193114492928,"internally_reserved":0,"allocated":39727104,"data_stored":204779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1512,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003721"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":6941,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:18.850547+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:18.850547+0000","last_peered":"2024-09-16T02:48:18.850547+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:18.850547+0000","last_undegraded":"2024-09-16T02:48:18.850547+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6042,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":964},{"shards":"2,3","objects":3021}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285298688,"data_stored":580518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640493,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86796,"kb_used_data":19516,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94285044,"statfs":{"total":96636764160,"available":96547885056,"internally_reserved":0,"allocated":19984384,"data_stored":102419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673197,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68512,"kb_used_data":19280,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94303328,"statfs":{"total":96636764160,"available":96566607872,"internally_reserved":0,"allocated":19742720,"data_stored":102360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986185,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19587072,"data_stored":17879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19767296,"data_stored":17923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:23.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:23.919 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:23.920 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:23.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2024-09-16T02:48:23.674370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":157504,"kb_used_data":40992,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188586176,"statfs":{"total":193273528320,"available":193112244224,"internally_reserved":0,"allocated":41975808,"data_stored":205328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1030,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":6941,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:18.850547+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:18.850547+0000","last_peered":"2024-09-16T02:48:18.850547+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:18.850547+0000","last_undegraded":"2024-09-16T02:48:18.850547+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6042,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":964},{"shards":"2,3","objects":3021}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":287502336,"data_stored":581056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640494,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87868,"kb_used_data":20588,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94283972,"statfs":{"total":96636764160,"available":96546787328,"internally_reserved":0,"allocated":21082112,"data_stored":102687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673198,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69636,"kb_used_data":20404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94302204,"statfs":{"total":96636764160,"available":96565456896,"internally_reserved":0,"allocated":20893696,"data_stored":102641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986186,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20684800,"data_stored":18147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20873216,"data_stored":18193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:23.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:24.284 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:24.284 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:24.305 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2024-09-16T02:48:23.674370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":157504,"kb_used_data":40992,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188586176,"statfs":{"total":193273528320,"available":193112244224,"internally_reserved":0,"allocated":41975808,"data_stored":205328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1030,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":6941,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:18.850547+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:18.850547+0000","last_peered":"2024-09-16T02:48:18.850547+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:18.850547+0000","last_undegraded":"2024-09-16T02:48:18.850547+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6042,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":964},{"shards":"2,3","objects":3021}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":287502336,"data_stored":581056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640494,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87868,"kb_used_data":20588,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94283972,"statfs":{"total":96636764160,"available":96546787328,"internally_reserved":0,"allocated":21082112,"data_stored":102687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673198,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69636,"kb_used_data":20404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94302204,"statfs":{"total":96636764160,"available":96565456896,"internally_reserved":0,"allocated":20893696,"data_stored":102641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986186,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20684800,"data_stored":18147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20873216,"data_stored":18193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 6941, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:18.850547+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:18.850547+0000', 'last_peered': '2024-09-16T02:48:18.850547+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:18.850547+0000', 'last_undegraded': '2024-09-16T02:48:18.850547+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6042, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 964, 'num_bytes_recovered': 964, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 964}, {'shards': '2,3', 'objects': 3021}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:24.306 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:24.307 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:24.307 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:24.307 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:24.307 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:24.307 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:24.307 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:24.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:24.711 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:24.711 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded","count":6},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1164,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":161284096,"bytes_avail":193112244224,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":30070,"misplaced_total":63846,"misplaced_ratio":0.47097703849888795,"recovering_objects_per_sec":42,"recovering_bytes_per_sec":42,"recovering_keys_per_sec":0,"num_objects_recovered":515,"num_bytes_recovered":515,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T02:45:37.631633+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:24.731 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:24.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:25.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:25.046 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:25.065 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2024-09-16T02:48:23.674370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":157504,"kb_used_data":40992,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188586176,"statfs":{"total":193273528320,"available":193112244224,"internally_reserved":0,"allocated":41975808,"data_stored":205328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1030,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":515,"num_bytes_recovered":515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":6941,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:18.850547+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:18.850547+0000","last_peered":"2024-09-16T02:48:18.850547+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:18.850547+0000","last_undegraded":"2024-09-16T02:48:18.850547+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6042,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":964,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":964},{"shards":"2,3","objects":3021}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":30070,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4004,"num_bytes_recovered":17121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":287502336,"data_stored":581056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640494,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87868,"kb_used_data":20588,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94283972,"statfs":{"total":96636764160,"available":96546787328,"internally_reserved":0,"allocated":21082112,"data_stored":102687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673198,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69636,"kb_used_data":20404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94302204,"statfs":{"total":96636764160,"available":96565456896,"internally_reserved":0,"allocated":20893696,"data_stored":102641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986186,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20684800,"data_stored":18147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20873216,"data_stored":18193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:28.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:28.404 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:28.423 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-09-16T02:48:27.675692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":29506,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4286,"num_bytes_recovered":17403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":163996,"kb_used_data":43324,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188579684,"statfs":{"total":193273528320,"available":193105596416,"internally_reserved":0,"allocated":44363776,"data_stored":205911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003513"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":7787,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:23.849788+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:23.849788+0000","last_peered":"2024-09-16T02:48:23.849788+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:23.849788+0000","last_undegraded":"2024-09-16T02:48:23.849788+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5478,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":1246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1246},{"shards":"2,3","objects":2739}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":29506,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4286,"num_bytes_recovered":17403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289865728,"data_stored":581633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640495,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93196,"kb_used_data":21756,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94278644,"statfs":{"total":96636764160,"available":96541331456,"internally_reserved":0,"allocated":22278144,"data_stored":102979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673199,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70800,"kb_used_data":21568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94301040,"statfs":{"total":96636764160,"available":96564264960,"internally_reserved":0,"allocated":22085632,"data_stored":102932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986187,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21856256,"data_stored":18433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22065152,"data_stored":18484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:28.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:28.760 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:28.760 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:28.781 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-09-16T02:48:27.675692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":29506,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4286,"num_bytes_recovered":17403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":163996,"kb_used_data":43324,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188579684,"statfs":{"total":193273528320,"available":193105596416,"internally_reserved":0,"allocated":44363776,"data_stored":205911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003513"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":7787,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:23.849788+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:23.849788+0000","last_peered":"2024-09-16T02:48:23.849788+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:23.849788+0000","last_undegraded":"2024-09-16T02:48:23.849788+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5478,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":1246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1246},{"shards":"2,3","objects":2739}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":29506,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4286,"num_bytes_recovered":17403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289865728,"data_stored":581633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640495,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93196,"kb_used_data":21756,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94278644,"statfs":{"total":96636764160,"available":96541331456,"internally_reserved":0,"allocated":22278144,"data_stored":102979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673199,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70800,"kb_used_data":21568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94301040,"statfs":{"total":96636764160,"available":96564264960,"internally_reserved":0,"allocated":22085632,"data_stored":102932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986187,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21856256,"data_stored":18433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22065152,"data_stored":18484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:28.782 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:28.782 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:28.782 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:28.782 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 7787, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:23.849788+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:23.849788+0000', 'last_peered': '2024-09-16T02:48:23.849788+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:23.849788+0000', 'last_undegraded': '2024-09-16T02:48:23.849788+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5478, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1246, 'num_bytes_recovered': 1246, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1246}, {'shards': '2,3', 'objects': 2739}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:28.782 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:28.782 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:28.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:28.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:29.185 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:29.185 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1168,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":167931904,"bytes_avail":193105596416,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":29506,"misplaced_total":63846,"misplaced_ratio":0.46214328227296941,"recovering_objects_per_sec":45,"recovering_bytes_per_sec":45,"recovering_keys_per_sec":0,"num_objects_recovered":552,"num_bytes_recovered":552,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T02:48:25.674905+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:48:29.206 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:29.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:29.547 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:29.568 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-09-16T02:48:27.675692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":29506,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4286,"num_bytes_recovered":17403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":163996,"kb_used_data":43324,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188579684,"statfs":{"total":193273528320,"available":193105596416,"internally_reserved":0,"allocated":44363776,"data_stored":205911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003513"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":7787,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:23.849788+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:23.849788+0000","last_peered":"2024-09-16T02:48:23.849788+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:23.849788+0000","last_undegraded":"2024-09-16T02:48:23.849788+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5478,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1246,"num_bytes_recovered":1246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1246},{"shards":"2,3","objects":2739}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":29506,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4286,"num_bytes_recovered":17403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289865728,"data_stored":581633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640495,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93196,"kb_used_data":21756,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94278644,"statfs":{"total":96636764160,"available":96541331456,"internally_reserved":0,"allocated":22278144,"data_stored":102979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673199,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70800,"kb_used_data":21568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94301040,"statfs":{"total":96636764160,"available":96564264960,"internally_reserved":0,"allocated":22085632,"data_stored":102932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986187,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21856256,"data_stored":18433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22065152,"data_stored":18484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:32.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:32.914 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:32.914 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:32.935 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2024-09-16T02:48:31.677095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28864,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":17724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":165256,"kb_used_data":44584,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188578424,"statfs":{"total":193273528320,"available":193104306176,"internally_reserved":0,"allocated":45654016,"data_stored":206226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003643"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":8750,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:28.851131+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:28.851131+0000","last_peered":"2024-09-16T02:48:28.851131+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:28.851131+0000","last_undegraded":"2024-09-16T02:48:28.851131+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4836,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":1567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1567},{"shards":"2,3","objects":2418}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28864,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":17724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":291155968,"data_stored":581948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640495,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93196,"kb_used_data":21756,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94278644,"statfs":{"total":96636764160,"available":96541331456,"internally_reserved":0,"allocated":22278144,"data_stored":102979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673200,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72060,"kb_used_data":22828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94299780,"statfs":{"total":96636764160,"available":96562974720,"internally_reserved":0,"allocated":23375872,"data_stored":103247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986188,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23146496,"data_stored":18748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22065152,"data_stored":18484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:32.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:33.270 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:33.270 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:33.293 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2024-09-16T02:48:31.677095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28864,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":17724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":165256,"kb_used_data":44584,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188578424,"statfs":{"total":193273528320,"available":193104306176,"internally_reserved":0,"allocated":45654016,"data_stored":206226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003643"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":8750,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:28.851131+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:28.851131+0000","last_peered":"2024-09-16T02:48:28.851131+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:28.851131+0000","last_undegraded":"2024-09-16T02:48:28.851131+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4836,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":1567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1567},{"shards":"2,3","objects":2418}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28864,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":17724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":291155968,"data_stored":581948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640495,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93196,"kb_used_data":21756,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94278644,"statfs":{"total":96636764160,"available":96541331456,"internally_reserved":0,"allocated":22278144,"data_stored":102979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673200,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72060,"kb_used_data":22828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94299780,"statfs":{"total":96636764160,"available":96562974720,"internally_reserved":0,"allocated":23375872,"data_stored":103247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986188,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23146496,"data_stored":18748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22065152,"data_stored":18484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 8750, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:28.851131+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:28.851131+0000', 'last_peered': '2024-09-16T02:48:28.851131+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:28.851131+0000', 'last_undegraded': '2024-09-16T02:48:28.851131+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4836, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1567, 'num_bytes_recovered': 1567, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1567}, {'shards': '2,3', 'objects': 2418}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:33.294 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:33.295 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:33.295 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:33.295 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:33.295 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:33.295 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:33.295 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:33.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:33.722 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:33.722 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1173,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":169222144,"bytes_avail":193104306176,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":28864,"misplaced_total":63846,"misplaced_ratio":0.45208783635623218,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":50,"recovering_keys_per_sec":0,"num_objects_recovered":603,"num_bytes_recovered":603,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T02:48:25.674905+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:48:33.744 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:33.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:34.083 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:34.083 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:34.103 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2024-09-16T02:48:33.677730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28864,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":17724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":166548,"kb_used_data":45876,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188577132,"statfs":{"total":193273528320,"available":193102983168,"internally_reserved":0,"allocated":46977024,"data_stored":206549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003907"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":8750,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:28.851131+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:28.851131+0000","last_peered":"2024-09-16T02:48:28.851131+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:28.851131+0000","last_undegraded":"2024-09-16T02:48:28.851131+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4836,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":1567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1567},{"shards":"2,3","objects":2418}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28864,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":17724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":292462592,"data_stored":582267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94488,"kb_used_data":23048,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94277352,"statfs":{"total":96636764160,"available":96540008448,"internally_reserved":0,"allocated":23601152,"data_stored":103302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673200,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72060,"kb_used_data":22828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94299780,"statfs":{"total":96636764160,"available":96562974720,"internally_reserved":0,"allocated":23375872,"data_stored":103247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986188,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23146496,"data_stored":18748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23371776,"data_stored":18803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:37.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:37.438 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:37.438 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:37.458 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2024-09-16T02:48:35.678338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28168,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4955,"num_bytes_recovered":18072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":166548,"kb_used_data":45876,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188577132,"statfs":{"total":193273528320,"available":193102983168,"internally_reserved":0,"allocated":46977024,"data_stored":206549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1902,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":951,"num_bytes_recovered":951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003968"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":9794,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:33.854935+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:33.854935+0000","last_peered":"2024-09-16T02:48:33.854935+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:33.854935+0000","last_undegraded":"2024-09-16T02:48:33.854935+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4140,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":1915,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1915},{"shards":"2,3","objects":2070}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28168,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4955,"num_bytes_recovered":18072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":292462592,"data_stored":582267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94488,"kb_used_data":23048,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94277352,"statfs":{"total":96636764160,"available":96540008448,"internally_reserved":0,"allocated":23601152,"data_stored":103302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673200,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72060,"kb_used_data":22828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94299780,"statfs":{"total":96636764160,"available":96562974720,"internally_reserved":0,"allocated":23375872,"data_stored":103247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986188,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23146496,"data_stored":18748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23371776,"data_stored":18803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:37.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:37.827 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:37.827 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:37.848 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2024-09-16T02:48:37.679267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28168,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4955,"num_bytes_recovered":18072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":173432,"kb_used_data":48600,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188570248,"statfs":{"total":193273528320,"available":193095933952,"internally_reserved":0,"allocated":49766400,"data_stored":207230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1338,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004503"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":9794,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:33.854935+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:33.854935+0000","last_peered":"2024-09-16T02:48:33.854935+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:33.854935+0000","last_undegraded":"2024-09-16T02:48:33.854935+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4140,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":1915,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1915},{"shards":"2,3","objects":2070}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28168,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4955,"num_bytes_recovered":18072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295346176,"data_stored":582971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95888,"kb_used_data":24448,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94275952,"statfs":{"total":96636764160,"available":96538574848,"internally_reserved":0,"allocated":25034752,"data_stored":103652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673201,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77544,"kb_used_data":24152,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94294296,"statfs":{"total":96636764160,"available":96557359104,"internally_reserved":0,"allocated":24731648,"data_stored":103578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986189,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24522752,"data_stored":19084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24879104,"data_stored":19171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:37.849 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:37.849 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:37.849 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 9794, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:33.854935+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:33.854935+0000', 'last_peered': '2024-09-16T02:48:33.854935+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:33.854935+0000', 'last_undegraded': '2024-09-16T02:48:33.854935+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4140, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1915, 'num_bytes_recovered': 1915, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1915}, {'shards': '2,3', 'objects': 2070}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:37.850 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:37.851 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:37.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:38.271 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:38.271 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1177,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":177594368,"bytes_avail":193095933952,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":28168,"misplaced_total":63846,"misplaced_ratio":0.44118660526892833,"recovering_objects_per_sec":55,"recovering_bytes_per_sec":55,"recovering_keys_per_sec":0,"num_objects_recovered":669,"num_bytes_recovered":669,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:38.292 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:38.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:38.630 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:38.630 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:38.652 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2024-09-16T02:48:37.679267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28168,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4955,"num_bytes_recovered":18072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":173432,"kb_used_data":48600,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188570248,"statfs":{"total":193273528320,"available":193095933952,"internally_reserved":0,"allocated":49766400,"data_stored":207230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1338,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004503"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":9794,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:33.854935+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:33.854935+0000","last_peered":"2024-09-16T02:48:33.854935+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:33.854935+0000","last_undegraded":"2024-09-16T02:48:33.854935+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4140,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":1915,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1915},{"shards":"2,3","objects":2070}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":28168,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4955,"num_bytes_recovered":18072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295346176,"data_stored":582971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95888,"kb_used_data":24448,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94275952,"statfs":{"total":96636764160,"available":96538574848,"internally_reserved":0,"allocated":25034752,"data_stored":103652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673201,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77544,"kb_used_data":24152,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94294296,"statfs":{"total":96636764160,"available":96557359104,"internally_reserved":0,"allocated":24731648,"data_stored":103578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986189,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24522752,"data_stored":19084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24879104,"data_stored":19171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:41.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:42.018 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:42.019 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:42.038 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-09-16T02:48:41.680389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":27412,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":18450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":174952,"kb_used_data":50120,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188568728,"statfs":{"total":193273528320,"available":193094377472,"internally_reserved":0,"allocated":51322880,"data_stored":207610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1452,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":726,"num_bytes_recovered":726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003855"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":10928,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:38.856415+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:38.856415+0000","last_peered":"2024-09-16T02:48:38.856415+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:38.856415+0000","last_undegraded":"2024-09-16T02:48:38.856415+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3384,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2293,"num_bytes_recovered":2293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2293},{"shards":"2,3","objects":1692}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":27412,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":18450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":296914944,"data_stored":583354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95888,"kb_used_data":24448,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94275952,"statfs":{"total":96636764160,"available":96538574848,"internally_reserved":0,"allocated":25034752,"data_stored":103652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79064,"kb_used_data":25672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292776,"statfs":{"total":96636764160,"available":96555802624,"internally_reserved":0,"allocated":26288128,"data_stored":103958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986190,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26091520,"data_stored":19467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24879104,"data_stored":19171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:42.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:42.401 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:42.402 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:42.422 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-09-16T02:48:41.680389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":27412,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":18450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":174952,"kb_used_data":50120,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188568728,"statfs":{"total":193273528320,"available":193094377472,"internally_reserved":0,"allocated":51322880,"data_stored":207610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1452,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":726,"num_bytes_recovered":726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003855"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":10928,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:38.856415+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:38.856415+0000","last_peered":"2024-09-16T02:48:38.856415+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:38.856415+0000","last_undegraded":"2024-09-16T02:48:38.856415+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3384,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2293,"num_bytes_recovered":2293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2293},{"shards":"2,3","objects":1692}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":27412,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":18450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":296914944,"data_stored":583354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95888,"kb_used_data":24448,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94275952,"statfs":{"total":96636764160,"available":96538574848,"internally_reserved":0,"allocated":25034752,"data_stored":103652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79064,"kb_used_data":25672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292776,"statfs":{"total":96636764160,"available":96555802624,"internally_reserved":0,"allocated":26288128,"data_stored":103958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986190,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26091520,"data_stored":19467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24879104,"data_stored":19171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:42.424 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:42.424 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:42.424 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:42.424 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 10928, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:38.856415+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:38.856415+0000', 'last_peered': '2024-09-16T02:48:38.856415+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:38.856415+0000', 'last_undegraded': '2024-09-16T02:48:38.856415+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3384, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2293, 'num_bytes_recovered': 2293, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2293}, {'shards': '2,3', 'objects': 1692}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:42.425 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:42.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:42.813 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:42.813 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1182,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":179150848,"bytes_avail":193094377472,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":27412,"misplaced_total":63846,"misplaced_ratio":0.42934561288099488,"recovering_objects_per_sec":60,"recovering_bytes_per_sec":60,"recovering_keys_per_sec":0,"num_objects_recovered":726,"num_bytes_recovered":726,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:42.832 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:42.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:43.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:43.166 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:43.183 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-09-16T02:48:41.680389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":27412,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":18450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":174952,"kb_used_data":50120,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188568728,"statfs":{"total":193273528320,"available":193094377472,"internally_reserved":0,"allocated":51322880,"data_stored":207610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1452,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":726,"num_bytes_recovered":726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003855"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":10928,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:38.856415+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:38.856415+0000","last_peered":"2024-09-16T02:48:38.856415+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:38.856415+0000","last_undegraded":"2024-09-16T02:48:38.856415+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3384,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2293,"num_bytes_recovered":2293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2293},{"shards":"2,3","objects":1692}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":27412,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":18450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":296914944,"data_stored":583354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640497,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95888,"kb_used_data":24448,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94275952,"statfs":{"total":96636764160,"available":96538574848,"internally_reserved":0,"allocated":25034752,"data_stored":103652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79064,"kb_used_data":25672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292776,"statfs":{"total":96636764160,"available":96555802624,"internally_reserved":0,"allocated":26288128,"data_stored":103958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986190,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26091520,"data_stored":19467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24879104,"data_stored":19171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:46.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:46.532 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:46.532 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:46.553 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-16T02:48:45.681490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":26558,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5760,"num_bytes_recovered":18877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":180768,"kb_used_data":51776,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188562912,"statfs":{"total":193273528320,"available":193088421888,"internally_reserved":0,"allocated":53018624,"data_stored":208024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2306,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":1153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003760"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":12209,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:43.856698+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:43.856698+0000","last_peered":"2024-09-16T02:48:43.856698+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:43.856698+0000","last_undegraded":"2024-09-16T02:48:43.856698+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2530,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2720,"num_bytes_recovered":2720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2720},{"shards":"2,3","objects":1265}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":26558,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5760,"num_bytes_recovered":18877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298651648,"data_stored":583778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640498,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101704,"kb_used_data":26104,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94270136,"statfs":{"total":96636764160,"available":96532619264,"internally_reserved":0,"allocated":26730496,"data_stored":104066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79064,"kb_used_data":25672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292776,"statfs":{"total":96636764160,"available":96555802624,"internally_reserved":0,"allocated":26288128,"data_stored":103958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986190,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26091520,"data_stored":19467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26615808,"data_stored":19595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:46.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:46.895 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:46.895 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:46.917 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-16T02:48:45.681490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":26558,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5760,"num_bytes_recovered":18877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":180768,"kb_used_data":51776,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188562912,"statfs":{"total":193273528320,"available":193088421888,"internally_reserved":0,"allocated":53018624,"data_stored":208024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2306,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":1153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003760"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":12209,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:43.856698+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:43.856698+0000","last_peered":"2024-09-16T02:48:43.856698+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:43.856698+0000","last_undegraded":"2024-09-16T02:48:43.856698+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2530,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2720,"num_bytes_recovered":2720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2720},{"shards":"2,3","objects":1265}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":26558,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5760,"num_bytes_recovered":18877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298651648,"data_stored":583778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640498,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101704,"kb_used_data":26104,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94270136,"statfs":{"total":96636764160,"available":96532619264,"internally_reserved":0,"allocated":26730496,"data_stored":104066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79064,"kb_used_data":25672,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94292776,"statfs":{"total":96636764160,"available":96555802624,"internally_reserved":0,"allocated":26288128,"data_stored":103958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986190,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26091520,"data_stored":19467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26615808,"data_stored":19595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:46.918 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:46.918 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:46.918 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:46.918 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 12209, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:43.856698+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:43.856698+0000', 'last_peered': '2024-09-16T02:48:43.856698+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:43.856698+0000', 'last_undegraded': '2024-09-16T02:48:43.856698+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2530, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2720, 'num_bytes_recovered': 2720, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2720}, {'shards': '2,3', 'objects': 1265}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:46.918 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:46.918 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:46.919 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:46.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:46.919 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:46.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:46.919 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:46.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:46.920 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:46.920 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:46.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:47.345 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:47.346 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1186,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":185106432,"bytes_avail":193088421888,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":26558,"misplaced_total":63846,"misplaced_ratio":0.41596967703536636,"recovering_objects_per_sec":96,"recovering_bytes_per_sec":96,"recovering_keys_per_sec":0,"num_objects_recovered":1153,"num_bytes_recovered":1153,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:47.367 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:47.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:47.709 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:47.709 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:47.728 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2024-09-16T02:48:47.682395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":26558,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5760,"num_bytes_recovered":18877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":184612,"kb_used_data":55620,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188559068,"statfs":{"total":193273528320,"available":193084485632,"internally_reserved":0,"allocated":56954880,"data_stored":208985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1610,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":805,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004056"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":12209,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:43.856698+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:43.856698+0000","last_peered":"2024-09-16T02:48:43.856698+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:43.856698+0000","last_undegraded":"2024-09-16T02:48:43.856698+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2530,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2720,"num_bytes_recovered":2720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2720},{"shards":"2,3","objects":1265}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":26558,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5760,"num_bytes_recovered":18877,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302706688,"data_stored":584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640499,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103768,"kb_used_data":28168,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94268072,"statfs":{"total":96636764160,"available":96530505728,"internally_reserved":0,"allocated":28844032,"data_stored":104582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673203,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80844,"kb_used_data":27452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94290996,"statfs":{"total":96636764160,"available":96553979904,"internally_reserved":0,"allocated":28110848,"data_stored":104403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986191,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28033024,"data_stored":19941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28729344,"data_stored":20111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:50.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:51.099 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:51.099 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:51.119 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-09-16T02:48:49.682783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":25492,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":19410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":184612,"kb_used_data":55620,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188559068,"statfs":{"total":193273528320,"available":193084485632,"internally_reserved":0,"allocated":56954880,"data_stored":208985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2676,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":1338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":13808,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:48.865959+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:48.865959+0000","last_peered":"2024-09-16T02:48:48.865959+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:48.865959+0000","last_undegraded":"2024-09-16T02:48:48.865959+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1464,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3253,"num_bytes_recovered":3253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3253},{"shards":"2,3","objects":732}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":25492,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":19410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302706688,"data_stored":584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640499,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103768,"kb_used_data":28168,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94268072,"statfs":{"total":96636764160,"available":96530505728,"internally_reserved":0,"allocated":28844032,"data_stored":104582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673203,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80844,"kb_used_data":27452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94290996,"statfs":{"total":96636764160,"available":96553979904,"internally_reserved":0,"allocated":28110848,"data_stored":104403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986191,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28033024,"data_stored":19941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28729344,"data_stored":20111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:51.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:51.468 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:51.489 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-09-16T02:48:49.682783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":25492,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":19410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":184612,"kb_used_data":55620,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188559068,"statfs":{"total":193273528320,"available":193084485632,"internally_reserved":0,"allocated":56954880,"data_stored":208985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2676,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":1338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":13808,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:48.865959+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:48.865959+0000","last_peered":"2024-09-16T02:48:48.865959+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:48.865959+0000","last_undegraded":"2024-09-16T02:48:48.865959+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1464,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3253,"num_bytes_recovered":3253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3253},{"shards":"2,3","objects":732}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":25492,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":19410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302706688,"data_stored":584768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640499,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103768,"kb_used_data":28168,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94268072,"statfs":{"total":96636764160,"available":96530505728,"internally_reserved":0,"allocated":28844032,"data_stored":104582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673203,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80844,"kb_used_data":27452,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94290996,"statfs":{"total":96636764160,"available":96553979904,"internally_reserved":0,"allocated":28110848,"data_stored":104403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986191,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28033024,"data_stored":19941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28729344,"data_stored":20111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:51.490 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:51.490 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 3995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.509887+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.496218+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3954, 'num_objects_misplaced': 3954, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3954}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:51.490 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:51.490 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 13808, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:48.865959+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:48.865959+0000', 'last_peered': '2024-09-16T02:48:48.865959+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:48.865959+0000', 'last_undegraded': '2024-09-16T02:48:48.865959+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1464, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3253, 'num_bytes_recovered': 3253, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3253}, {'shards': '2,3', 'objects': 732}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4120, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818918+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:00.818918+0000', 'last_peered': '2024-09-16T02:48:00.818918+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:00.818918+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818907+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:00.818907+0000', 'last_peered': '2024-09-16T02:48:00.818907+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:00.818907+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4077, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818878+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:00.818878+0000', 'last_peered': '2024-09-16T02:48:00.818878+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:00.818878+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.820076+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:00.820076+0000', 'last_peered': '2024-09-16T02:48:00.820076+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:00.820076+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:51.491 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:00.818826+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:00.818826+0000', 'last_peered': '2024-09-16T02:48:00.818826+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:00.818826+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:51.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:51.918 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:51.918 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24028/63846 objects degraded (37.634%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1191,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":195870720,"bytes_avail":193077657600,"bytes_total":193273528320,"degraded_objects":24028,"degraded_total":63846,"degraded_ratio":0.37634307552548324,"misplaced_objects":25492,"misplaced_total":63846,"misplaced_ratio":0.3992732512608464,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":79,"recovering_keys_per_sec":0,"num_objects_recovered":960,"num_bytes_recovered":960,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:51.940 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:51.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:52.286 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:52.287 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:52.308 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":621,"stamp":"2024-09-16T02:48:51.683413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":25492,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":19410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":191280,"kb_used_data":58128,"kb_used_omap":3,"kb_used_meta":133116,"kb_avail":188552400,"statfs":{"total":193273528320,"available":193077657600,"internally_reserved":0,"allocated":59523072,"data_stored":209612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":136311690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1920,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":960,"num_bytes_recovered":960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003665"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.509887+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.496218+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3954,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3954}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":13808,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:48.865959+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:48.865959+0000","last_peered":"2024-09-16T02:48:48.865959+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:48.865959+0000","last_undegraded":"2024-09-16T02:48:48.865959+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1464,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3253,"num_bytes_recovered":3253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3253},{"shards":"2,3","objects":732}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818918+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:00.818918+0000","last_peered":"2024-09-16T02:48:00.818918+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:00.818918+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818907+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:00.818907+0000","last_peered":"2024-09-16T02:48:00.818907+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:00.818907+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4077,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818878+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:00.818878+0000","last_peered":"2024-09-16T02:48:00.818878+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:00.818878+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.820076+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:00.820076+0000","last_peered":"2024-09-16T02:48:00.820076+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:00.820076+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:00.818826+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:00.818826+0000","last_peered":"2024-09-16T02:48:00.818826+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:00.818826+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24028,"num_objects_misplaced":25492,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":19410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":305463296,"data_stored":585441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640499,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103768,"kb_used_data":28168,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94268072,"statfs":{"total":96636764160,"available":96530505728,"internally_reserved":0,"allocated":28844032,"data_stored":104582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673204,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87512,"kb_used_data":29960,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94284328,"statfs":{"total":96636764160,"available":96547151872,"internally_reserved":0,"allocated":30679040,"data_stored":105030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986192,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30789632,"data_stored":20614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28729344,"data_stored":20111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:55.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:55.636 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:55.637 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:55.656 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2024-09-16T02:48:54.760719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":29374,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":19446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":181948,"kb_used_data":61276,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188561732,"statfs":{"total":193273528320,"available":193087213568,"internally_reserved":0,"allocated":62746624,"data_stored":210399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":2816,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":569,"num_bytes_recovered":569,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.079229"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4106,"reported_epoch":34,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:53.861519+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:53.861519+0000","last_peered":"2024-09-16T02:48:53.861519+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:53.861519+0000","last_undegraded":"2024-09-16T02:48:53.861519+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7836,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":36,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":36},{"shards":"2,3","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":13808,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:48.865959+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:48.865959+0000","last_peered":"2024-09-16T02:48:48.865959+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:48.865959+0000","last_undegraded":"2024-09-16T02:48:48.865959+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1464,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3253,"num_bytes_recovered":3253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3253},{"shards":"2,3","objects":732}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4122,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764757+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:53.764757+0000","last_peered":"2024-09-16T02:48:53.764757+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:53.764757+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4022,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764369+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:53.764369+0000","last_peered":"2024-09-16T02:48:53.764369+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:53.764369+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4079,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764294+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:53.764294+0000","last_peered":"2024-09-16T02:48:53.764294+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:53.764294+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4047,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764396+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:53.764396+0000","last_peered":"2024-09-16T02:48:53.764396+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:53.764396+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4083,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764199+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:53.764199+0000","last_peered":"2024-09-16T02:48:53.764199+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:53.764199+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":29374,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":19446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":308797440,"data_stored":586255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640500,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94436,"kb_used_data":31316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94277404,"statfs":{"total":96636764160,"available":96540061696,"internally_reserved":0,"allocated":32067584,"data_stored":105369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673204,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87512,"kb_used_data":29960,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94284328,"statfs":{"total":96636764160,"available":96547151872,"internally_reserved":0,"allocated":30679040,"data_stored":105030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986192,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30789632,"data_stored":20614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32063488,"data_stored":20925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:55.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:55.985 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:55.986 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:56.010 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2024-09-16T02:48:55.684523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":29374,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":19446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":181948,"kb_used_data":61276,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188561732,"statfs":{"total":193273528320,"available":193087213568,"internally_reserved":0,"allocated":62746624,"data_stored":210399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":2816,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":569,"num_bytes_recovered":569,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002128"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4106,"reported_epoch":34,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:53.861519+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:53.861519+0000","last_peered":"2024-09-16T02:48:53.861519+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:53.861519+0000","last_undegraded":"2024-09-16T02:48:53.861519+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7836,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":36,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":36},{"shards":"2,3","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":13808,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:48.865959+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:48.865959+0000","last_peered":"2024-09-16T02:48:48.865959+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:48.865959+0000","last_undegraded":"2024-09-16T02:48:48.865959+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1464,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3253,"num_bytes_recovered":3253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3253},{"shards":"2,3","objects":732}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4122,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764757+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:53.764757+0000","last_peered":"2024-09-16T02:48:53.764757+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:53.764757+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4022,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764369+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:53.764369+0000","last_peered":"2024-09-16T02:48:53.764369+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:53.764369+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4079,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764294+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:53.764294+0000","last_peered":"2024-09-16T02:48:53.764294+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:53.764294+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4047,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764396+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:53.764396+0000","last_peered":"2024-09-16T02:48:53.764396+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:53.764396+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4083,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764199+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:53.764199+0000","last_peered":"2024-09-16T02:48:53.764199+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:53.764199+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":29374,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":19446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":308797440,"data_stored":586255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640500,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94436,"kb_used_data":31316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94277404,"statfs":{"total":96636764160,"available":96540061696,"internally_reserved":0,"allocated":32067584,"data_stored":105369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673204,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87512,"kb_used_data":29960,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94284328,"statfs":{"total":96636764160,"available":96547151872,"internally_reserved":0,"allocated":30679040,"data_stored":105030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986192,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30789632,"data_stored":20614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32063488,"data_stored":20925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 4106, 'reported_epoch': 34, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:53.861519+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:48:53.861519+0000', 'last_peered': '2024-09-16T02:48:53.861519+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:53.861519+0000', 'last_undegraded': '2024-09-16T02:48:53.861519+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7836, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 36, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 36}, {'shards': '2,3', 'objects': 3918}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "19'3985", 'reported_seq': 13808, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:48.865959+0000', 'last_change': '2024-09-16T02:47:59.674492+0000', 'last_active': '2024-09-16T02:48:48.865959+0000', 'last_peered': '2024-09-16T02:48:48.865959+0000', 'last_clean': '2024-09-16T02:46:36.687660+0000', 'last_became_active': '2024-09-16T02:47:26.500788+0000', 'last_became_peered': '2024-09-16T02:47:26.500788+0000', 'last_unstale': '2024-09-16T02:48:48.865959+0000', 'last_undegraded': '2024-09-16T02:48:48.865959+0000', 'last_fullsized': '2024-09-16T02:47:26.334211+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3985, 'log_dups_size': 0, 'ondisk_log_size': 3985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001483763, 'stat_sum': {'num_bytes': 56385, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 7970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1464, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3985, 'num_write_kb': 4033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3253, 'num_bytes_recovered': 3253, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3253}, {'shards': '2,3', 'objects': 732}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4122, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:53.764757+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:53.764757+0000', 'last_peered': '2024-09-16T02:48:53.764757+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:53.764757+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4022, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:53.764369+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:53.764369+0000', 'last_peered': '2024-09-16T02:48:53.764369+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:53.764369+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:48:56.011 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4079, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:53.764294+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:53.764294+0000', 'last_peered': '2024-09-16T02:48:53.764294+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:53.764294+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:56.012 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:48:56.012 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4047, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:53.764396+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:53.764396+0000', 'last_peered': '2024-09-16T02:48:53.764396+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:53.764396+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:56.012 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:48:56.012 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4083, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:53.764199+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:53.764199+0000', 'last_peered': '2024-09-16T02:48:53.764199+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:53.764199+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:48:56.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:48:56.423 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:56.423 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 7 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1196,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+remapped","count":2},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":186314752,"bytes_avail":193087213568,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":29374,"misplaced_total":63846,"misplaced_ratio":0.46007580741158416,"recovering_objects_per_sec":71,"recovering_bytes_per_sec":71,"recovering_keys_per_sec":0,"num_objects_recovered":569,"num_bytes_recovered":569,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:48:56.444 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:48:56.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:48:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:48:56.812 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:48:56.831 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2024-09-16T02:48:55.684523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":29374,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":19446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":181948,"kb_used_data":61276,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188561732,"statfs":{"total":193273528320,"available":193087213568,"internally_reserved":0,"allocated":62746624,"data_stored":210399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":2816,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":569,"num_bytes_recovered":569,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002128"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4106,"reported_epoch":34,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:53.861519+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:53.861519+0000","last_peered":"2024-09-16T02:48:53.861519+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:53.861519+0000","last_undegraded":"2024-09-16T02:48:53.861519+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7836,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":36,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":36},{"shards":"2,3","objects":3918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":13808,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:48.865959+0000","last_change":"2024-09-16T02:47:59.674492+0000","last_active":"2024-09-16T02:48:48.865959+0000","last_peered":"2024-09-16T02:48:48.865959+0000","last_clean":"2024-09-16T02:46:36.687660+0000","last_became_active":"2024-09-16T02:47:26.500788+0000","last_became_peered":"2024-09-16T02:47:26.500788+0000","last_unstale":"2024-09-16T02:48:48.865959+0000","last_undegraded":"2024-09-16T02:48:48.865959+0000","last_fullsized":"2024-09-16T02:47:26.334211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1464,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3253,"num_bytes_recovered":3253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3253},{"shards":"2,3","objects":732}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4122,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764757+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:53.764757+0000","last_peered":"2024-09-16T02:48:53.764757+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:53.764757+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4022,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764369+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:53.764369+0000","last_peered":"2024-09-16T02:48:53.764369+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:53.764369+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4079,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764294+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:53.764294+0000","last_peered":"2024-09-16T02:48:53.764294+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:53.764294+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4047,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764396+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:53.764396+0000","last_peered":"2024-09-16T02:48:53.764396+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:53.764396+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T02:48:00.806620+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:00.806620+0000","last_peered":"2024-09-16T02:48:00.806620+0000","last_clean":"2024-09-16T02:48:00.806620+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:00.806620+0000","last_undegraded":"2024-09-16T02:48:00.806620+0000","last_fullsized":"2024-09-16T02:48:00.806620+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4083,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:53.764199+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:53.764199+0000","last_peered":"2024-09-16T02:48:53.764199+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:53.764199+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":29374,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":19446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":308797440,"data_stored":586255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640500,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94436,"kb_used_data":31316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94277404,"statfs":{"total":96636764160,"available":96540061696,"internally_reserved":0,"allocated":32067584,"data_stored":105369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673204,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87512,"kb_used_data":29960,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94284328,"statfs":{"total":96636764160,"available":96547151872,"internally_reserved":0,"allocated":30679040,"data_stored":105030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986192,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30789632,"data_stored":20614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32063488,"data_stored":20925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114958336,"data_stored":263853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:48:59.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:00.171 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:00.172 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:00.192 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-16T02:48:59.685642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":173556,"kb_used_data":65364,"kb_used_omap":3,"kb_used_meta":108156,"kb_avail":188570124,"statfs":{"total":193273528320,"available":193095806976,"internally_reserved":0,"allocated":66932736,"data_stored":327367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":110752650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":1938,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1008,"num_bytes_recovered":53408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002229"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4828,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:58.862304+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:58.862304+0000","last_peered":"2024-09-16T02:48:58.862304+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:58.862304+0000","last_undegraded":"2024-09-16T02:48:58.862304+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7356,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":276},{"shards":"2,3","objects":3678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":296042496,"data_stored":635529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640501,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95956,"kb_used_data":32836,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94275884,"statfs":{"total":96636764160,"available":96538505216,"internally_reserved":0,"allocated":33624064,"data_stored":163722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673205,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77600,"kb_used_data":32528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94294240,"statfs":{"total":96636764160,"available":96557301760,"internally_reserved":0,"allocated":33308672,"data_stored":163645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986193,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33120256,"data_stored":73571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33349632,"data_stored":73627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:00.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:00.508 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:00.508 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:00.528 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-16T02:48:59.685642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":173556,"kb_used_data":65364,"kb_used_omap":3,"kb_used_meta":108156,"kb_avail":188570124,"statfs":{"total":193273528320,"available":193095806976,"internally_reserved":0,"allocated":66932736,"data_stored":327367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":110752650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":1938,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1008,"num_bytes_recovered":53408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002229"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4828,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:58.862304+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:58.862304+0000","last_peered":"2024-09-16T02:48:58.862304+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:58.862304+0000","last_undegraded":"2024-09-16T02:48:58.862304+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7356,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":276},{"shards":"2,3","objects":3678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":296042496,"data_stored":635529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640501,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95956,"kb_used_data":32836,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94275884,"statfs":{"total":96636764160,"available":96538505216,"internally_reserved":0,"allocated":33624064,"data_stored":163722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673205,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77600,"kb_used_data":32528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94294240,"statfs":{"total":96636764160,"available":96557301760,"internally_reserved":0,"allocated":33308672,"data_stored":163645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986193,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33120256,"data_stored":73571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33349632,"data_stored":73627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:00.529 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:00.529 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 4828, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:58.862304+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:48:58.862304+0000', 'last_peered': '2024-09-16T02:48:58.862304+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:58.862304+0000', 'last_undegraded': '2024-09-16T02:48:58.862304+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7356, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 276, 'num_bytes_recovered': 276, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 276}, {'shards': '2,3', 'objects': 3678}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:00.529 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:00.529 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779678+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.779678+0000', 'last_peered': '2024-09-16T02:48:54.779678+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.779678+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4024, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.773104+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.773104+0000', 'last_peered': '2024-09-16T02:48:54.773104+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.773104+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771897+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.771897+0000', 'last_peered': '2024-09-16T02:48:54.771897+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.771897+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771702+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.771702+0000', 'last_peered': '2024-09-16T02:48:54.771702+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.771702+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:00.530 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771671+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.771671+0000', 'last_peered': '2024-09-16T02:48:54.771671+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.771671+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:00.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:00.958 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:00.958 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1200,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":177721344,"bytes_avail":193095806976,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":27430,"misplaced_total":63846,"misplaced_ratio":0.4296275412711838,"recovering_objects_per_sec":125,"recovering_bytes_per_sec":6674,"recovering_keys_per_sec":0,"num_objects_recovered":1008,"num_bytes_recovered":53408,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:00.981 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:00.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:01.348 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:01.348 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:01.369 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-16T02:48:59.685642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":173556,"kb_used_data":65364,"kb_used_omap":3,"kb_used_meta":108156,"kb_avail":188570124,"statfs":{"total":193273528320,"available":193095806976,"internally_reserved":0,"allocated":66932736,"data_stored":327367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":110752650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":1938,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1008,"num_bytes_recovered":53408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002229"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4828,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:58.862304+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:58.862304+0000","last_peered":"2024-09-16T02:48:58.862304+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:58.862304+0000","last_undegraded":"2024-09-16T02:48:58.862304+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7356,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":276},{"shards":"2,3","objects":3678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":296042496,"data_stored":635529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640501,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95956,"kb_used_data":32836,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94275884,"statfs":{"total":96636764160,"available":96538505216,"internally_reserved":0,"allocated":33624064,"data_stored":163722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673205,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77600,"kb_used_data":32528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94294240,"statfs":{"total":96636764160,"available":96557301760,"internally_reserved":0,"allocated":33308672,"data_stored":163645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986193,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33120256,"data_stored":73571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33349632,"data_stored":73627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:04.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:04.707 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:04.708 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:04.728 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2024-09-16T02:49:03.686984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":179764,"kb_used_data":67412,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188563916,"statfs":{"total":193273528320,"available":193089449984,"internally_reserved":0,"allocated":69029888,"data_stored":327879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":1938,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1008,"num_bytes_recovered":53408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.939077"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4828,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:58.862304+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:58.862304+0000","last_peered":"2024-09-16T02:48:58.862304+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:58.862304+0000","last_undegraded":"2024-09-16T02:48:58.862304+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7356,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":276},{"shards":"2,3","objects":3678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298090496,"data_stored":636029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640502,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101100,"kb_used_data":33820,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94270740,"statfs":{"total":96636764160,"available":96533237760,"internally_reserved":0,"allocated":34631680,"data_stored":163968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78664,"kb_used_data":33592,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94293176,"statfs":{"total":96636764160,"available":96556212224,"internally_reserved":0,"allocated":34398208,"data_stored":163911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986194,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34168832,"data_stored":73827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34349056,"data_stored":73871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:04.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:05.044 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:05.045 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:05.064 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2024-09-16T02:49:03.686984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":179764,"kb_used_data":67412,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188563916,"statfs":{"total":193273528320,"available":193089449984,"internally_reserved":0,"allocated":69029888,"data_stored":327879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3954,"num_objects_misplaced":1938,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1008,"num_bytes_recovered":53408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.939077"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4828,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:48:58.862304+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:48:58.862304+0000","last_peered":"2024-09-16T02:48:58.862304+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:48:58.862304+0000","last_undegraded":"2024-09-16T02:48:58.862304+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7356,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":276},{"shards":"2,3","objects":3678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":27430,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":72818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298090496,"data_stored":636029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640502,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101100,"kb_used_data":33820,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94270740,"statfs":{"total":96636764160,"available":96533237760,"internally_reserved":0,"allocated":34631680,"data_stored":163968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78664,"kb_used_data":33592,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94293176,"statfs":{"total":96636764160,"available":96556212224,"internally_reserved":0,"allocated":34398208,"data_stored":163911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986194,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34168832,"data_stored":73827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34349056,"data_stored":73871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 4828, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:48:58.862304+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:48:58.862304+0000', 'last_peered': '2024-09-16T02:48:58.862304+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:48:58.862304+0000', 'last_undegraded': '2024-09-16T02:48:58.862304+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7356, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 276, 'num_bytes_recovered': 276, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 276}, {'shards': '2,3', 'objects': 3678}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779678+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.779678+0000', 'last_peered': '2024-09-16T02:48:54.779678+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.779678+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4024, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.773104+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.773104+0000', 'last_peered': '2024-09-16T02:48:54.773104+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.773104+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:05.065 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771897+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.771897+0000', 'last_peered': '2024-09-16T02:48:54.771897+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.771897+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:05.066 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:05.066 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771702+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.771702+0000', 'last_peered': '2024-09-16T02:48:54.771702+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.771702+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:05.066 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:05.066 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771671+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.771671+0000', 'last_peered': '2024-09-16T02:48:54.771671+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.771671+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:05.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:05.459 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:05.459 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1205,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":184078336,"bytes_avail":193089449984,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":27430,"misplaced_total":63846,"misplaced_ratio":0.4296275412711838,"recovering_objects_per_sec":101,"recovering_bytes_per_sec":5373,"recovering_keys_per_sec":0,"num_objects_recovered":1008,"num_bytes_recovered":53408,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:05.480 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:05.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:05.806 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:05.807 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:05.826 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2024-09-16T02:49:05.687570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26936,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7548,"num_bytes_recovered":73065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":179764,"kb_used_data":67412,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188563916,"statfs":{"total":193273528320,"available":193089449984,"internally_reserved":0,"allocated":69029888,"data_stored":327879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2438,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1219,"num_bytes_recovered":53619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.926851"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":5569,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:03.877044+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:03.877044+0000","last_peered":"2024-09-16T02:49:03.877044+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:03.877044+0000","last_undegraded":"2024-09-16T02:49:03.877044+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6862,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":523,"num_bytes_recovered":523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":523},{"shards":"2,3","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26936,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7548,"num_bytes_recovered":73065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":298090496,"data_stored":636029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640502,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101100,"kb_used_data":33820,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94270740,"statfs":{"total":96636764160,"available":96533237760,"internally_reserved":0,"allocated":34631680,"data_stored":163968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673206,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78664,"kb_used_data":33592,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94293176,"statfs":{"total":96636764160,"available":96556212224,"internally_reserved":0,"allocated":34398208,"data_stored":163911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986194,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34168832,"data_stored":73827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34349056,"data_stored":73871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:08.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:09.330 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:09.330 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:09.350 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2024-09-16T02:49:07.688150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26936,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7548,"num_bytes_recovered":73065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":181904,"kb_used_data":69552,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188561776,"statfs":{"total":193273528320,"available":193087258624,"internally_reserved":0,"allocated":71221248,"data_stored":328414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2438,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1219,"num_bytes_recovered":53619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":5569,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:03.877044+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:03.877044+0000","last_peered":"2024-09-16T02:49:03.877044+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:03.877044+0000","last_undegraded":"2024-09-16T02:49:03.877044+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6862,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":523,"num_bytes_recovered":523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":523},{"shards":"2,3","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26936,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7548,"num_bytes_recovered":73065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300249088,"data_stored":636556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640503,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102148,"kb_used_data":34868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94269692,"statfs":{"total":96636764160,"available":96532164608,"internally_reserved":0,"allocated":35704832,"data_stored":164230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673207,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79756,"kb_used_data":34684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94292084,"statfs":{"total":96636764160,"available":96555094016,"internally_reserved":0,"allocated":35516416,"data_stored":164184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986195,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35250176,"data_stored":74091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":74134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:09.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:09.683 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:09.703 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2024-09-16T02:49:07.688150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26936,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7548,"num_bytes_recovered":73065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":181904,"kb_used_data":69552,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188561776,"statfs":{"total":193273528320,"available":193087258624,"internally_reserved":0,"allocated":71221248,"data_stored":328414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2438,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1219,"num_bytes_recovered":53619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":5569,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:03.877044+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:03.877044+0000","last_peered":"2024-09-16T02:49:03.877044+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:03.877044+0000","last_undegraded":"2024-09-16T02:49:03.877044+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6862,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":523,"num_bytes_recovered":523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":523},{"shards":"2,3","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26936,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7548,"num_bytes_recovered":73065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300249088,"data_stored":636556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640503,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102148,"kb_used_data":34868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94269692,"statfs":{"total":96636764160,"available":96532164608,"internally_reserved":0,"allocated":35704832,"data_stored":164230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673207,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79756,"kb_used_data":34684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94292084,"statfs":{"total":96636764160,"available":96555094016,"internally_reserved":0,"allocated":35516416,"data_stored":164184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986195,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35250176,"data_stored":74091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":74134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:09.704 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:09.704 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 5569, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:03.877044+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:03.877044+0000', 'last_peered': '2024-09-16T02:49:03.877044+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:03.877044+0000', 'last_undegraded': '2024-09-16T02:49:03.877044+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6862, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 523, 'num_bytes_recovered': 523, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 523}, {'shards': '2,3', 'objects': 3431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779678+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.779678+0000', 'last_peered': '2024-09-16T02:48:54.779678+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.779678+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4024, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.773104+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.773104+0000', 'last_peered': '2024-09-16T02:48:54.773104+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.773104+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771897+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.771897+0000', 'last_peered': '2024-09-16T02:48:54.771897+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.771897+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771702+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.771702+0000', 'last_peered': '2024-09-16T02:48:54.771702+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.771702+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:09.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771671+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.771671+0000', 'last_peered': '2024-09-16T02:48:54.771671+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.771671+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:09.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:10.117 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:10.117 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1209,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":186269696,"bytes_avail":193087258624,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":26402,"misplaced_total":63846,"misplaced_ratio":0.41352629765372922,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":62,"recovering_keys_per_sec":0,"num_objects_recovered":754,"num_bytes_recovered":754,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:10.138 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:10.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:10.481 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:10.481 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:10.500 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":632,"stamp":"2024-09-16T02:49:09.688758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":181904,"kb_used_data":69552,"kb_used_omap":3,"kb_used_meta":112316,"kb_avail":188561776,"statfs":{"total":193273528320,"available":193087258624,"internally_reserved":0,"allocated":71221248,"data_stored":328414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":115012490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1508,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":754,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003787"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":6370,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:08.888391+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:08.888391+0000","last_peered":"2024-09-16T02:49:08.888391+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:08.888391+0000","last_undegraded":"2024-09-16T02:49:08.888391+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6328,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":790},{"shards":"2,3","objects":3164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":300249088,"data_stored":636556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640503,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102148,"kb_used_data":34868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94269692,"statfs":{"total":96636764160,"available":96532164608,"internally_reserved":0,"allocated":35704832,"data_stored":164230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673207,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79756,"kb_used_data":34684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94292084,"statfs":{"total":96636764160,"available":96555094016,"internally_reserved":0,"allocated":35516416,"data_stored":164184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986195,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35250176,"data_stored":74091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":74134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:13.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:13.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:13.869 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2024-09-16T02:49:13.689909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":188236,"kb_used_data":71724,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188555444,"statfs":{"total":193273528320,"available":193080774656,"internally_reserved":0,"allocated":73445376,"data_stored":328957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1028,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":514,"num_bytes_recovered":514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":6370,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:08.888391+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:08.888391+0000","last_peered":"2024-09-16T02:49:08.888391+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:08.888391+0000","last_undegraded":"2024-09-16T02:49:08.888391+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6328,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":790},{"shards":"2,3","objects":3164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302473216,"data_stored":637099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103244,"kb_used_data":35964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94268596,"statfs":{"total":96636764160,"available":96531042304,"internally_reserved":0,"allocated":36827136,"data_stored":164504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673208,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84992,"kb_used_data":35760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94286848,"statfs":{"total":96636764160,"available":96549732352,"internally_reserved":0,"allocated":36618240,"data_stored":164453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986196,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36356096,"data_stored":74361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36544512,"data_stored":74407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:13.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:14.199 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:14.200 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:14.212 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2024-09-16T02:49:13.689909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":188236,"kb_used_data":71724,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188555444,"statfs":{"total":193273528320,"available":193080774656,"internally_reserved":0,"allocated":73445376,"data_stored":328957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1028,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":514,"num_bytes_recovered":514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":6370,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:08.888391+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:08.888391+0000","last_peered":"2024-09-16T02:49:08.888391+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:08.888391+0000","last_undegraded":"2024-09-16T02:49:08.888391+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6328,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":790},{"shards":"2,3","objects":3164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302473216,"data_stored":637099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103244,"kb_used_data":35964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94268596,"statfs":{"total":96636764160,"available":96531042304,"internally_reserved":0,"allocated":36827136,"data_stored":164504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673208,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84992,"kb_used_data":35760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94286848,"statfs":{"total":96636764160,"available":96549732352,"internally_reserved":0,"allocated":36618240,"data_stored":164453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986196,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36356096,"data_stored":74361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36544512,"data_stored":74407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:14.213 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:14.213 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 6370, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:08.888391+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:08.888391+0000', 'last_peered': '2024-09-16T02:49:08.888391+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:08.888391+0000', 'last_undegraded': '2024-09-16T02:49:08.888391+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6328, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 790, 'num_bytes_recovered': 790, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 790}, {'shards': '2,3', 'objects': 3164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:14.213 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:14.213 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779678+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.779678+0000', 'last_peered': '2024-09-16T02:48:54.779678+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.779678+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:14.213 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:14.214 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4024, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.773104+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.773104+0000', 'last_peered': '2024-09-16T02:48:54.773104+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.773104+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:14.214 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:14.214 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771897+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.771897+0000', 'last_peered': '2024-09-16T02:48:54.771897+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.771897+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:14.214 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:14.214 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771702+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.771702+0000', 'last_peered': '2024-09-16T02:48:54.771702+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.771702+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:14.214 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:14.215 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771671+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.771671+0000', 'last_peered': '2024-09-16T02:48:54.771671+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.771671+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:14.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:14.839 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:14.839 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1214,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":192753664,"bytes_avail":193080774656,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":26402,"misplaced_total":63846,"misplaced_ratio":0.41352629765372922,"recovering_objects_per_sec":42,"recovering_bytes_per_sec":42,"recovering_keys_per_sec":0,"num_objects_recovered":514,"num_bytes_recovered":514,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:14.860 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:14.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:15.190 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:15.190 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:15.210 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2024-09-16T02:49:13.689909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":188236,"kb_used_data":71724,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188555444,"statfs":{"total":193273528320,"available":193080774656,"internally_reserved":0,"allocated":73445376,"data_stored":328957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1028,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":514,"num_bytes_recovered":514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":6370,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:08.888391+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:08.888391+0000","last_peered":"2024-09-16T02:49:08.888391+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:08.888391+0000","last_undegraded":"2024-09-16T02:49:08.888391+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6328,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":790},{"shards":"2,3","objects":3164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3972,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.762899+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.762899+0000","last_peered":"2024-09-16T02:48:54.762899+0000","last_clean":"2024-09-16T02:48:54.762899+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.762899+0000","last_undegraded":"2024-09-16T02:48:54.762899+0000","last_fullsized":"2024-09-16T02:48:54.762899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":26402,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":73332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302473216,"data_stored":637099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103244,"kb_used_data":35964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94268596,"statfs":{"total":96636764160,"available":96531042304,"internally_reserved":0,"allocated":36827136,"data_stored":164504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673208,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84992,"kb_used_data":35760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94286848,"statfs":{"total":96636764160,"available":96549732352,"internally_reserved":0,"allocated":36618240,"data_stored":164453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986196,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36356096,"data_stored":74361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36544512,"data_stored":74407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:18.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:18.568 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:18.569 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:18.588 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2024-09-16T02:49:17.691054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25814,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8109,"num_bytes_recovered":73626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":190576,"kb_used_data":74064,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188553104,"statfs":{"total":193273528320,"available":193078378496,"internally_reserved":0,"allocated":75841536,"data_stored":329542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":561,"num_bytes_recovered":561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003484"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":7252,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:13.904107+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:13.904107+0000","last_peered":"2024-09-16T02:49:13.904107+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:13.904107+0000","last_undegraded":"2024-09-16T02:49:13.904107+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5740,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1084,"num_bytes_recovered":1084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1084},{"shards":"2,3","objects":2870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25814,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8109,"num_bytes_recovered":73626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":304861184,"data_stored":637682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640505,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104420,"kb_used_data":37140,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94267420,"statfs":{"total":96636764160,"available":96529838080,"internally_reserved":0,"allocated":38031360,"data_stored":164798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673209,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86156,"kb_used_data":36924,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94285684,"statfs":{"total":96636764160,"available":96548540416,"internally_reserved":0,"allocated":37810176,"data_stored":164744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986197,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37543936,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37744640,"data_stored":74700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:18.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:18.915 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:18.915 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:18.935 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2024-09-16T02:49:17.691054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25814,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8109,"num_bytes_recovered":73626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":190576,"kb_used_data":74064,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188553104,"statfs":{"total":193273528320,"available":193078378496,"internally_reserved":0,"allocated":75841536,"data_stored":329542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":561,"num_bytes_recovered":561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003484"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":7252,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:13.904107+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:13.904107+0000","last_peered":"2024-09-16T02:49:13.904107+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:13.904107+0000","last_undegraded":"2024-09-16T02:49:13.904107+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5740,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1084,"num_bytes_recovered":1084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1084},{"shards":"2,3","objects":2870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25814,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8109,"num_bytes_recovered":73626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":304861184,"data_stored":637682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640505,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104420,"kb_used_data":37140,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94267420,"statfs":{"total":96636764160,"available":96529838080,"internally_reserved":0,"allocated":38031360,"data_stored":164798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673209,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86156,"kb_used_data":36924,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94285684,"statfs":{"total":96636764160,"available":96548540416,"internally_reserved":0,"allocated":37810176,"data_stored":164744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986197,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37543936,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37744640,"data_stored":74700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:18.936 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:18.936 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 7252, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:13.904107+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:13.904107+0000', 'last_peered': '2024-09-16T02:49:13.904107+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:13.904107+0000', 'last_undegraded': '2024-09-16T02:49:13.904107+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5740, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1084, 'num_bytes_recovered': 1084, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1084}, {'shards': '2,3', 'objects': 2870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:18.936 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779678+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.779678+0000', 'last_peered': '2024-09-16T02:48:54.779678+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.779678+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4024, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.773104+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.773104+0000', 'last_peered': '2024-09-16T02:48:54.773104+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.773104+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771897+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.771897+0000', 'last_peered': '2024-09-16T02:48:54.771897+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.771897+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771702+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.771702+0000', 'last_peered': '2024-09-16T02:48:54.771702+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.771702+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:18.937 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771671+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.771671+0000', 'last_peered': '2024-09-16T02:48:54.771671+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.771671+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:18.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:19.341 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:19.341 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1218,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":195149824,"bytes_avail":193078378496,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":25814,"misplaced_total":63846,"misplaced_ratio":0.40431663690755881,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":46,"recovering_keys_per_sec":0,"num_objects_recovered":561,"num_bytes_recovered":561,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:19.363 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:19.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:19.681 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:19.681 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:19.701 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2024-09-16T02:49:17.691054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25814,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8109,"num_bytes_recovered":73626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":190576,"kb_used_data":74064,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188553104,"statfs":{"total":193273528320,"available":193078378496,"internally_reserved":0,"allocated":75841536,"data_stored":329542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":561,"num_bytes_recovered":561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003484"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":7252,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:13.904107+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:13.904107+0000","last_peered":"2024-09-16T02:49:13.904107+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:13.904107+0000","last_undegraded":"2024-09-16T02:49:13.904107+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5740,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1084,"num_bytes_recovered":1084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1084},{"shards":"2,3","objects":2870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4042,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.774687+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.774687+0000","last_peered":"2024-09-16T02:48:54.774687+0000","last_clean":"2024-09-16T02:48:54.774687+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.774687+0000","last_undegraded":"2024-09-16T02:48:54.774687+0000","last_fullsized":"2024-09-16T02:48:54.774687+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25814,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8109,"num_bytes_recovered":73626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":304861184,"data_stored":637682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640505,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104420,"kb_used_data":37140,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94267420,"statfs":{"total":96636764160,"available":96529838080,"internally_reserved":0,"allocated":38031360,"data_stored":164798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673209,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86156,"kb_used_data":36924,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94285684,"statfs":{"total":96636764160,"available":96548540416,"internally_reserved":0,"allocated":37810176,"data_stored":164744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986197,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37543936,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37744640,"data_stored":74700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:22.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:23.037 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:23.037 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:23.057 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2024-09-16T02:49:21.692223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25198,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8417,"num_bytes_recovered":73934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":191784,"kb_used_data":75272,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188551896,"statfs":{"total":193273528320,"available":193077141504,"internally_reserved":0,"allocated":77078528,"data_stored":329844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1204,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003465"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":8176,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:18.913335+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:18.913335+0000","last_peered":"2024-09-16T02:49:18.913335+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:18.913335+0000","last_undegraded":"2024-09-16T02:49:18.913335+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5124,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1392,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1392},{"shards":"2,3","objects":2562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25198,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8417,"num_bytes_recovered":73934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":306106368,"data_stored":637986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640505,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104420,"kb_used_data":37140,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94267420,"statfs":{"total":96636764160,"available":96529838080,"internally_reserved":0,"allocated":38031360,"data_stored":164798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673210,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87364,"kb_used_data":38132,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94284476,"statfs":{"total":96636764160,"available":96547303424,"internally_reserved":0,"allocated":39047168,"data_stored":165046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986198,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38789120,"data_stored":74955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37744640,"data_stored":74700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:23.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:23.408 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2024-09-16T02:49:21.692223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25198,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8417,"num_bytes_recovered":73934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":191784,"kb_used_data":75272,"kb_used_omap":3,"kb_used_meta":116476,"kb_avail":188551896,"statfs":{"total":193273528320,"available":193077141504,"internally_reserved":0,"allocated":77078528,"data_stored":329844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":119272330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1204,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003465"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":8176,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:18.913335+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:18.913335+0000","last_peered":"2024-09-16T02:49:18.913335+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:18.913335+0000","last_undegraded":"2024-09-16T02:49:18.913335+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5124,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1392,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1392},{"shards":"2,3","objects":2562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25198,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8417,"num_bytes_recovered":73934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":306106368,"data_stored":637986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640505,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104420,"kb_used_data":37140,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94267420,"statfs":{"total":96636764160,"available":96529838080,"internally_reserved":0,"allocated":38031360,"data_stored":164798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673210,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87364,"kb_used_data":38132,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94284476,"statfs":{"total":96636764160,"available":96547303424,"internally_reserved":0,"allocated":39047168,"data_stored":165046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986198,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38789120,"data_stored":74955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37744640,"data_stored":74700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:23.409 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:23.409 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 8176, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:18.913335+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:18.913335+0000', 'last_peered': '2024-09-16T02:49:18.913335+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:18.913335+0000', 'last_undegraded': '2024-09-16T02:49:18.913335+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5124, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1392, 'num_bytes_recovered': 1392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1392}, {'shards': '2,3', 'objects': 2562}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:23.409 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:23.409 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779678+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.779678+0000', 'last_peered': '2024-09-16T02:48:54.779678+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.779678+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:23.409 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:23.409 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4024, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.773104+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.773104+0000', 'last_peered': '2024-09-16T02:48:54.773104+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.773104+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:23.410 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:23.410 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771897+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.771897+0000', 'last_peered': '2024-09-16T02:48:54.771897+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.771897+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:23.410 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:23.410 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771702+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.771702+0000', 'last_peered': '2024-09-16T02:48:54.771702+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.771702+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:23.410 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:23.410 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771671+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.771671+0000', 'last_peered': '2024-09-16T02:48:54.771671+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.771671+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:23.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:23.829 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:23.829 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1223,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":196386816,"bytes_avail":193077141504,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":25198,"misplaced_total":63846,"misplaced_ratio":0.39466842088776116,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":50,"recovering_keys_per_sec":0,"num_objects_recovered":602,"num_bytes_recovered":602,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:23.852 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:23.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:24.197 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:24.197 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:24.216 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-16T02:49:23.692839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25198,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8417,"num_bytes_recovered":73934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":197176,"kb_used_data":76504,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188546504,"statfs":{"total":193273528320,"available":193071620096,"internally_reserved":0,"allocated":78340096,"data_stored":330152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1204,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003542"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":8176,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:18.913335+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:18.913335+0000","last_peered":"2024-09-16T02:49:18.913335+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:18.913335+0000","last_undegraded":"2024-09-16T02:49:18.913335+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5124,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1392,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1392},{"shards":"2,3","objects":2562}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771671+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.771671+0000","last_peered":"2024-09-16T02:48:54.771671+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.771671+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":25198,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8417,"num_bytes_recovered":73934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":307367936,"data_stored":638294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640506,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109812,"kb_used_data":38372,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94262028,"statfs":{"total":96636764160,"available":96524316672,"internally_reserved":0,"allocated":39292928,"data_stored":165106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673210,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87364,"kb_used_data":38132,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94284476,"statfs":{"total":96636764160,"available":96547303424,"internally_reserved":0,"allocated":39047168,"data_stored":165046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986198,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38789120,"data_stored":74955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39006208,"data_stored":75008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:27.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:27.569 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:27.570 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:27.591 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2024-09-16T02:49:25.693340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":24512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8760,"num_bytes_recovered":74277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":197176,"kb_used_data":76504,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188546504,"statfs":{"total":193273528320,"available":193071620096,"internally_reserved":0,"allocated":78340096,"data_stored":330152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1890,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":945,"num_bytes_recovered":945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003430"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":9205,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:23.922355+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:23.922355+0000","last_peered":"2024-09-16T02:49:23.922355+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:23.922355+0000","last_undegraded":"2024-09-16T02:49:23.922355+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4438,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":1735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1735},{"shards":"2,3","objects":2219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":24512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8760,"num_bytes_recovered":74277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":307367936,"data_stored":638294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640506,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109812,"kb_used_data":38372,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94262028,"statfs":{"total":96636764160,"available":96524316672,"internally_reserved":0,"allocated":39292928,"data_stored":165106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673210,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87364,"kb_used_data":38132,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94284476,"statfs":{"total":96636764160,"available":96547303424,"internally_reserved":0,"allocated":39047168,"data_stored":165046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986198,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38789120,"data_stored":74955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39006208,"data_stored":75008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:27.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:27.917 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:27.917 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:27.938 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":641,"stamp":"2024-09-16T02:49:27.693988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":24512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8760,"num_bytes_recovered":74277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":199880,"kb_used_data":79208,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188543800,"statfs":{"total":193273528320,"available":193068851200,"internally_reserved":0,"allocated":81108992,"data_stored":330828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1302,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":651,"num_bytes_recovered":651,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003698"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":9205,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:23.922355+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:23.922355+0000","last_peered":"2024-09-16T02:49:23.922355+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:23.922355+0000","last_undegraded":"2024-09-16T02:49:23.922355+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4438,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":1735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1735},{"shards":"2,3","objects":2219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":24512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8760,"num_bytes_recovered":74277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310202368,"data_stored":638986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640507,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111180,"kb_used_data":39740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94260660,"statfs":{"total":96636764160,"available":96522915840,"internally_reserved":0,"allocated":40693760,"data_stored":165448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673211,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88700,"kb_used_data":39468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94283140,"statfs":{"total":96636764160,"available":96545935360,"internally_reserved":0,"allocated":40415232,"data_stored":165380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986199,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40153088,"data_stored":75288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40476672,"data_stored":75367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:27.939 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:27.939 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 9205, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:23.922355+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:23.922355+0000', 'last_peered': '2024-09-16T02:49:23.922355+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:23.922355+0000', 'last_undegraded': '2024-09-16T02:49:23.922355+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4438, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1735, 'num_bytes_recovered': 1735, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1735}, {'shards': '2,3', 'objects': 2219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:27.939 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779678+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.779678+0000', 'last_peered': '2024-09-16T02:48:54.779678+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.779678+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4024, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.773104+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.773104+0000', 'last_peered': '2024-09-16T02:48:54.773104+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.773104+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771897+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.771897+0000', 'last_peered': '2024-09-16T02:48:54.771897+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.771897+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.771702+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.771702+0000', 'last_peered': '2024-09-16T02:48:54.771702+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.771702+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:27.940 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4086, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778635+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.778635+0000', 'last_peered': '2024-09-16T02:48:54.778635+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.778635+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:27.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:28.347 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:28.347 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1227,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":204677120,"bytes_avail":193068851200,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":24512,"misplaced_total":63846,"misplaced_ratio":0.38392381668389564,"recovering_objects_per_sec":54,"recovering_bytes_per_sec":54,"recovering_keys_per_sec":0,"num_objects_recovered":651,"num_bytes_recovered":651,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:28.367 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:28.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:28.692 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:28.692 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:28.710 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":641,"stamp":"2024-09-16T02:49:27.693988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":24512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8760,"num_bytes_recovered":74277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":199880,"kb_used_data":79208,"kb_used_omap":3,"kb_used_meta":120636,"kb_avail":188543800,"statfs":{"total":193273528320,"available":193068851200,"internally_reserved":0,"allocated":81108992,"data_stored":330828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":123532170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1302,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":651,"num_bytes_recovered":651,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003698"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":9205,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:23.922355+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:23.922355+0000","last_peered":"2024-09-16T02:49:23.922355+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:23.922355+0000","last_undegraded":"2024-09-16T02:49:23.922355+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4438,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":1735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1735},{"shards":"2,3","objects":2219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779678+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.779678+0000","last_peered":"2024-09-16T02:48:54.779678+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.779678+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4024,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.773104+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.773104+0000","last_peered":"2024-09-16T02:48:54.773104+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.773104+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771897+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.771897+0000","last_peered":"2024-09-16T02:48:54.771897+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.771897+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.771702+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.771702+0000","last_peered":"2024-09-16T02:48:54.771702+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.771702+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":24512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8760,"num_bytes_recovered":74277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310202368,"data_stored":638986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640507,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111180,"kb_used_data":39740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94260660,"statfs":{"total":96636764160,"available":96522915840,"internally_reserved":0,"allocated":40693760,"data_stored":165448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673211,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88700,"kb_used_data":39468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94283140,"statfs":{"total":96636764160,"available":96545935360,"internally_reserved":0,"allocated":40415232,"data_stored":165380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986199,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40153088,"data_stored":75288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40476672,"data_stored":75367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:31.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:32.091 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:32.092 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:32.113 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2024-09-16T02:49:31.695282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":23742,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9145,"num_bytes_recovered":74662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":205548,"kb_used_data":80716,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188538132,"statfs":{"total":193273528320,"available":193063047168,"internally_reserved":0,"allocated":82653184,"data_stored":331205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003548"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":10360,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:28.928645+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:28.928645+0000","last_peered":"2024-09-16T02:49:28.928645+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:28.928645+0000","last_undegraded":"2024-09-16T02:49:28.928645+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3668,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2120,"num_bytes_recovered":2120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2120},{"shards":"2,3","objects":1834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":23742,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9145,"num_bytes_recovered":74662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311767040,"data_stored":639368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640507,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111180,"kb_used_data":39740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94260660,"statfs":{"total":96636764160,"available":96522915840,"internally_reserved":0,"allocated":40693760,"data_stored":165448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94368,"kb_used_data":40976,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94277472,"statfs":{"total":96636764160,"available":96540131328,"internally_reserved":0,"allocated":41959424,"data_stored":165757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986200,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41717760,"data_stored":75670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40476672,"data_stored":75367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:32.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:32.449 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:32.470 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2024-09-16T02:49:31.695282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":23742,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9145,"num_bytes_recovered":74662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":205548,"kb_used_data":80716,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188538132,"statfs":{"total":193273528320,"available":193063047168,"internally_reserved":0,"allocated":82653184,"data_stored":331205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003548"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":10360,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:28.928645+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:28.928645+0000","last_peered":"2024-09-16T02:49:28.928645+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:28.928645+0000","last_undegraded":"2024-09-16T02:49:28.928645+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3668,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2120,"num_bytes_recovered":2120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2120},{"shards":"2,3","objects":1834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":23742,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9145,"num_bytes_recovered":74662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311767040,"data_stored":639368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640507,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111180,"kb_used_data":39740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94260660,"statfs":{"total":96636764160,"available":96522915840,"internally_reserved":0,"allocated":40693760,"data_stored":165448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94368,"kb_used_data":40976,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94277472,"statfs":{"total":96636764160,"available":96540131328,"internally_reserved":0,"allocated":41959424,"data_stored":165757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986200,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41717760,"data_stored":75670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40476672,"data_stored":75367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 10360, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:28.928645+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:28.928645+0000', 'last_peered': '2024-09-16T02:49:28.928645+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:28.928645+0000', 'last_undegraded': '2024-09-16T02:49:28.928645+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3668, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2120, 'num_bytes_recovered': 2120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2120}, {'shards': '2,3', 'objects': 1834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.784900+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.784900+0000', 'last_peered': '2024-09-16T02:48:54.784900+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.784900+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4025, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778960+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.778960+0000', 'last_peered': '2024-09-16T02:48:54.778960+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.778960+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4082, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778139+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.778139+0000', 'last_peered': '2024-09-16T02:48:54.778139+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.778139+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:32.471 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779035+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.779035+0000', 'last_peered': '2024-09-16T02:48:54.779035+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.779035+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:32.472 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:32.472 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4086, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778635+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.778635+0000', 'last_peered': '2024-09-16T02:48:54.778635+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.778635+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:32.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:32.908 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:32.908 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1232,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":210481152,"bytes_avail":193063047168,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":23742,"misplaced_total":63846,"misplaced_ratio":0.37186354665914856,"recovering_objects_per_sec":60,"recovering_bytes_per_sec":60,"recovering_keys_per_sec":0,"num_objects_recovered":728,"num_bytes_recovered":728,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:32.930 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:32.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:33.283 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:33.284 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:33.306 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2024-09-16T02:49:31.695282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":23742,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9145,"num_bytes_recovered":74662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":205548,"kb_used_data":80716,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188538132,"statfs":{"total":193273528320,"available":193063047168,"internally_reserved":0,"allocated":82653184,"data_stored":331205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003548"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":10360,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:28.928645+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:28.928645+0000","last_peered":"2024-09-16T02:49:28.928645+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:28.928645+0000","last_undegraded":"2024-09-16T02:49:28.928645+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3668,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2120,"num_bytes_recovered":2120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2120},{"shards":"2,3","objects":1834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":23742,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9145,"num_bytes_recovered":74662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311767040,"data_stored":639368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640507,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111180,"kb_used_data":39740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94260660,"statfs":{"total":96636764160,"available":96522915840,"internally_reserved":0,"allocated":40693760,"data_stored":165448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94368,"kb_used_data":40976,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94277472,"statfs":{"total":96636764160,"available":96540131328,"internally_reserved":0,"allocated":41959424,"data_stored":165757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986200,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41717760,"data_stored":75670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40476672,"data_stored":75367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:36.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:36.672 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:36.692 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2024-09-16T02:49:35.696495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":22926,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9553,"num_bytes_recovered":75070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":207180,"kb_used_data":82348,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188536500,"statfs":{"total":193273528320,"available":193061376000,"internally_reserved":0,"allocated":84324352,"data_stored":331613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2272,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":1136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003656"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":11585,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:33.937161+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:33.937161+0000","last_peered":"2024-09-16T02:49:33.937161+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:33.937161+0000","last_undegraded":"2024-09-16T02:49:33.937161+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2852,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2528,"num_bytes_recovered":2528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2528},{"shards":"2,3","objects":1426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":22926,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9553,"num_bytes_recovered":75070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313495552,"data_stored":639790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640508,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112812,"kb_used_data":41372,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94259028,"statfs":{"total":96636764160,"available":96521244672,"internally_reserved":0,"allocated":42364928,"data_stored":165856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94368,"kb_used_data":40976,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94277472,"statfs":{"total":96636764160,"available":96540131328,"internally_reserved":0,"allocated":41959424,"data_stored":165757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986200,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41717760,"data_stored":75670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42205184,"data_stored":75789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:36.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:37.014 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:37.015 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:37.035 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2024-09-16T02:49:35.696495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":22926,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9553,"num_bytes_recovered":75070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":207180,"kb_used_data":82348,"kb_used_omap":3,"kb_used_meta":124796,"kb_avail":188536500,"statfs":{"total":193273528320,"available":193061376000,"internally_reserved":0,"allocated":84324352,"data_stored":331613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":127792010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2272,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":1136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003656"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":11585,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:33.937161+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:33.937161+0000","last_peered":"2024-09-16T02:49:33.937161+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:33.937161+0000","last_undegraded":"2024-09-16T02:49:33.937161+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2852,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2528,"num_bytes_recovered":2528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2528},{"shards":"2,3","objects":1426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":22926,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9553,"num_bytes_recovered":75070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313495552,"data_stored":639790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640508,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112812,"kb_used_data":41372,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94259028,"statfs":{"total":96636764160,"available":96521244672,"internally_reserved":0,"allocated":42364928,"data_stored":165856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94368,"kb_used_data":40976,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94277472,"statfs":{"total":96636764160,"available":96540131328,"internally_reserved":0,"allocated":41959424,"data_stored":165757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986200,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41717760,"data_stored":75670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42205184,"data_stored":75789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:37.036 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:37.036 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 11585, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:33.937161+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:33.937161+0000', 'last_peered': '2024-09-16T02:49:33.937161+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:33.937161+0000', 'last_undegraded': '2024-09-16T02:49:33.937161+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2852, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2528, 'num_bytes_recovered': 2528, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2528}, {'shards': '2,3', 'objects': 1426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:37.036 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:37.036 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.784900+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.784900+0000', 'last_peered': '2024-09-16T02:48:54.784900+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.784900+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:37.036 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:37.037 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4025, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778960+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.778960+0000', 'last_peered': '2024-09-16T02:48:54.778960+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.778960+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:37.037 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:37.037 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4082, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778139+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.778139+0000', 'last_peered': '2024-09-16T02:48:54.778139+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.778139+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:37.037 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:37.037 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779035+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.779035+0000', 'last_peered': '2024-09-16T02:48:54.779035+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.779035+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:37.037 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:37.037 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4086, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778635+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.778635+0000', 'last_peered': '2024-09-16T02:48:54.778635+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.778635+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:37.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:37.444 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:37.444 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1237,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":212152320,"bytes_avail":193061376000,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":22926,"misplaced_total":63846,"misplaced_ratio":0.35908279297058548,"recovering_objects_per_sec":94,"recovering_bytes_per_sec":94,"recovering_keys_per_sec":0,"num_objects_recovered":1136,"num_bytes_recovered":1136,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:37.466 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:37.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:37.802 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:37.802 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:37.822 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2024-09-16T02:49:37.697271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":22926,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9553,"num_bytes_recovered":75070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":215056,"kb_used_data":86064,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188528624,"statfs":{"total":193273528320,"available":193053310976,"internally_reserved":0,"allocated":88129536,"data_stored":332542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1586,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":793,"num_bytes_recovered":793,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003930"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":11585,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:33.937161+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:33.937161+0000","last_peered":"2024-09-16T02:49:33.937161+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:33.937161+0000","last_undegraded":"2024-09-16T02:49:33.937161+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2852,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2528,"num_bytes_recovered":2528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2528},{"shards":"2,3","objects":1426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":22926,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9553,"num_bytes_recovered":75070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317358080,"data_stored":640733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118952,"kb_used_data":43352,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94252888,"statfs":{"total":96636764160,"available":96514957312,"internally_reserved":0,"allocated":44392448,"data_stored":166351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673213,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96104,"kb_used_data":42712,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94275736,"statfs":{"total":96636764160,"available":96538353664,"internally_reserved":0,"allocated":43737088,"data_stored":166191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986201,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43540480,"data_stored":76115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44244992,"data_stored":76287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:40.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:41.182 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:41.183 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:41.203 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2024-09-16T02:49:39.697876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":21918,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10057,"num_bytes_recovered":75574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":215056,"kb_used_data":86064,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188528624,"statfs":{"total":193273528320,"available":193053310976,"internally_reserved":0,"allocated":88129536,"data_stored":332542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2594,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":1297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003887"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":13097,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:38.942371+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:38.942371+0000","last_peered":"2024-09-16T02:49:38.942371+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:38.942371+0000","last_undegraded":"2024-09-16T02:49:38.942371+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1844,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3032,"num_bytes_recovered":3032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3032},{"shards":"2,3","objects":922}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":21918,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10057,"num_bytes_recovered":75574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317358080,"data_stored":640733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118952,"kb_used_data":43352,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94252888,"statfs":{"total":96636764160,"available":96514957312,"internally_reserved":0,"allocated":44392448,"data_stored":166351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673213,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96104,"kb_used_data":42712,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94275736,"statfs":{"total":96636764160,"available":96538353664,"internally_reserved":0,"allocated":43737088,"data_stored":166191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986201,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43540480,"data_stored":76115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44244992,"data_stored":76287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:41.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:41.556 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:41.556 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:41.576 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2024-09-16T02:49:39.697876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":21918,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10057,"num_bytes_recovered":75574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":215056,"kb_used_data":86064,"kb_used_omap":3,"kb_used_meta":128956,"kb_avail":188528624,"statfs":{"total":193273528320,"available":193053310976,"internally_reserved":0,"allocated":88129536,"data_stored":332542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":132051850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2594,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1297,"num_bytes_recovered":1297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003887"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":13097,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:38.942371+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:38.942371+0000","last_peered":"2024-09-16T02:49:38.942371+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:38.942371+0000","last_undegraded":"2024-09-16T02:49:38.942371+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1844,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3032,"num_bytes_recovered":3032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3032},{"shards":"2,3","objects":922}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":21918,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10057,"num_bytes_recovered":75574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317358080,"data_stored":640733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118952,"kb_used_data":43352,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94252888,"statfs":{"total":96636764160,"available":96514957312,"internally_reserved":0,"allocated":44392448,"data_stored":166351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673213,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96104,"kb_used_data":42712,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94275736,"statfs":{"total":96636764160,"available":96538353664,"internally_reserved":0,"allocated":43737088,"data_stored":166191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986201,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43540480,"data_stored":76115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44244992,"data_stored":76287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:41.577 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:41.577 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 13097, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:38.942371+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:38.942371+0000', 'last_peered': '2024-09-16T02:49:38.942371+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:38.942371+0000', 'last_undegraded': '2024-09-16T02:49:38.942371+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1844, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3032, 'num_bytes_recovered': 3032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3032}, {'shards': '2,3', 'objects': 922}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:41.577 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:41.577 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.784900+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.784900+0000', 'last_peered': '2024-09-16T02:48:54.784900+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.784900+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:41.577 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:41.578 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4025, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778960+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.778960+0000', 'last_peered': '2024-09-16T02:48:54.778960+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.778960+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:41.578 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:41.578 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4082, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778139+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.778139+0000', 'last_peered': '2024-09-16T02:48:54.778139+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.778139+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:41.578 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:41.578 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779035+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.779035+0000', 'last_peered': '2024-09-16T02:48:54.779035+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.779035+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:41.578 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:41.578 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4086, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778635+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.778635+0000', 'last_peered': '2024-09-16T02:48:54.778635+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.778635+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:41.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:42.003 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:42.003 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1241,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":245301248,"bytes_avail":193028227072,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":21918,"misplaced_total":63846,"misplaced_ratio":0.34329480312000754,"recovering_objects_per_sec":75,"recovering_bytes_per_sec":75,"recovering_keys_per_sec":0,"num_objects_recovered":912,"num_bytes_recovered":912,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:42.024 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:42.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:42.369 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:42.390 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2024-09-16T02:49:41.698484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":21918,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10057,"num_bytes_recovered":75574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":239552,"kb_used_data":88352,"kb_used_omap":3,"kb_used_meta":151164,"kb_avail":188504128,"statfs":{"total":193273528320,"available":193028227072,"internally_reserved":0,"allocated":90472448,"data_stored":333114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":154792842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1824,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004067"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":13097,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:38.942371+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:38.942371+0000","last_peered":"2024-09-16T02:49:38.942371+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:38.942371+0000","last_undegraded":"2024-09-16T02:49:38.942371+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1844,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3032,"num_bytes_recovered":3032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3032},{"shards":"2,3","objects":922}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":21918,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10057,"num_bytes_recovered":75574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":319823872,"data_stored":641335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118952,"kb_used_data":43352,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94252888,"statfs":{"total":96636764160,"available":96514957312,"internally_reserved":0,"allocated":44392448,"data_stored":166351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673214,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120600,"kb_used_data":45000,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94251240,"statfs":{"total":96636764160,"available":96513269760,"internally_reserved":0,"allocated":46080000,"data_stored":166763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986202,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46006272,"data_stored":76717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44244992,"data_stored":76287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:45.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:45.724 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:45.725 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:45.745 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":650,"stamp":"2024-09-16T02:49:45.699683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":20512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10760,"num_bytes_recovered":76277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":260272,"kb_used_data":91024,"kb_used_omap":3,"kb_used_meta":169212,"kb_avail":188483408,"statfs":{"total":193273528320,"available":193007009792,"internally_reserved":0,"allocated":93208576,"data_stored":333782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":173273994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3230,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1615,"num_bytes_recovered":1615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003775"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":15206,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:43.944686+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:43.944686+0000","last_peered":"2024-09-16T02:49:43.944686+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:43.944686+0000","last_undegraded":"2024-09-16T02:49:43.944686+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":438,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3735,"num_bytes_recovered":3735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3735},{"shards":"2,3","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":20512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10760,"num_bytes_recovered":76277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":322564096,"data_stored":642004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640510,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139672,"kb_used_data":46024,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94232168,"statfs":{"total":96636764160,"available":96493740032,"internally_reserved":0,"allocated":47128576,"data_stored":167019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673214,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120600,"kb_used_data":45000,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94251240,"statfs":{"total":96636764160,"available":96513269760,"internally_reserved":0,"allocated":46080000,"data_stored":166763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986202,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46006272,"data_stored":76717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46985216,"data_stored":76956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:45.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:46.081 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:46.082 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:46.101 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2024-09-16T02:49:45.787225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":20512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10760,"num_bytes_recovered":76277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":260272,"kb_used_data":91024,"kb_used_omap":3,"kb_used_meta":169212,"kb_avail":188483408,"statfs":{"total":193273528320,"available":193007009792,"internally_reserved":0,"allocated":93208576,"data_stored":333782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":173273994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2414,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1207,"num_bytes_recovered":1207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.090730"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":15206,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:43.944686+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:43.944686+0000","last_peered":"2024-09-16T02:49:43.944686+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:43.944686+0000","last_undegraded":"2024-09-16T02:49:43.944686+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":438,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3735,"num_bytes_recovered":3735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3735},{"shards":"2,3","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4043,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.776093+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:48:54.776093+0000","last_peered":"2024-09-16T02:48:54.776093+0000","last_clean":"2024-09-16T02:48:54.776093+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:48:54.776093+0000","last_undegraded":"2024-09-16T02:48:54.776093+0000","last_fullsized":"2024-09-16T02:48:54.776093+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3973,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T02:48:54.763049+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:48:54.763049+0000","last_peered":"2024-09-16T02:48:54.763049+0000","last_clean":"2024-09-16T02:48:54.763049+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:48:54.763049+0000","last_undegraded":"2024-09-16T02:48:54.763049+0000","last_fullsized":"2024-09-16T02:48:54.763049+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":20512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10760,"num_bytes_recovered":76277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":322564096,"data_stored":642004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640510,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139672,"kb_used_data":46024,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94232168,"statfs":{"total":96636764160,"available":96493740032,"internally_reserved":0,"allocated":47128576,"data_stored":167019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673214,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120600,"kb_used_data":45000,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94251240,"statfs":{"total":96636764160,"available":96513269760,"internally_reserved":0,"allocated":46080000,"data_stored":166763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986202,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46006272,"data_stored":76717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46985216,"data_stored":76956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:46.102 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-16T02:49:46.102 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "19'3954", 'reported_seq': 15206, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:43.944686+0000', 'last_change': '2024-09-16T02:48:52.860865+0000', 'last_active': '2024-09-16T02:49:43.944686+0000', 'last_peered': '2024-09-16T02:49:43.944686+0000', 'last_clean': '2024-09-16T02:46:35.253185+0000', 'last_became_active': '2024-09-16T02:47:26.504510+0000', 'last_became_peered': '2024-09-16T02:47:26.504510+0000', 'last_unstale': '2024-09-16T02:49:43.944686+0000', 'last_undegraded': '2024-09-16T02:49:43.944686+0000', 'last_fullsized': '2024-09-16T02:47:26.334503+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3954, 'log_dups_size': 0, 'ondisk_log_size': 3954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3954, 'num_objects': 3954, 'num_object_clones': 0, 'num_object_copies': 7908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 438, 'num_objects_unfound': 0, 'num_objects_dirty': 3954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3954, 'num_write_kb': 3954, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3735, 'num_bytes_recovered': 3735, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3735}, {'shards': '2,3', 'objects': 219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:46.102 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.784900+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:48:54.784900+0000', 'last_peered': '2024-09-16T02:48:54.784900+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:48:54.784900+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4025, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778960+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:48:54.778960+0000', 'last_peered': '2024-09-16T02:48:54.778960+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:48:54.778960+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4082, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778139+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:48:54.778139+0000', 'last_peered': '2024-09-16T02:48:54.778139+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:48:54.778139+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4050, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.779035+0000', 'last_change': '2024-09-16T02:47:26.521997+0000', 'last_active': '2024-09-16T02:48:54.779035+0000', 'last_peered': '2024-09-16T02:48:54.779035+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:48:54.779035+0000', 'last_undegraded': '2024-09-16T02:47:26.504474+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3997, 'num_objects_misplaced': 3997, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:46.103 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4086, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:48:54.778635+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:48:54.778635+0000', 'last_peered': '2024-09-16T02:48:54.778635+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:48:54.778635+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:46.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:46.498 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20074/63846 objects degraded (31.441%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1246,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":266518528,"bytes_avail":193007009792,"bytes_total":193273528320,"degraded_objects":20074,"degraded_total":63846,"degraded_ratio":0.31441280581398989,"misplaced_objects":20512,"misplaced_total":63846,"misplaced_ratio":0.32127306330858629,"recovering_objects_per_sec":134,"recovering_bytes_per_sec":134,"recovering_keys_per_sec":0,"num_objects_recovered":1615,"num_bytes_recovered":1615,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:46.519 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:46.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:46.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:46.879 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:46.902 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2024-09-16T02:49:46.788984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":20512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10760,"num_bytes_recovered":76277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":267440,"kb_used_data":94032,"kb_used_omap":3,"kb_used_meta":173372,"kb_avail":188476240,"statfs":{"total":193273528320,"available":192999669760,"internally_reserved":0,"allocated":96288768,"data_stored":337330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":177533834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2414,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1207,"num_bytes_recovered":1207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.091713"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":15206,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:43.944686+0000","last_change":"2024-09-16T02:48:52.860865+0000","last_active":"2024-09-16T02:49:43.944686+0000","last_peered":"2024-09-16T02:49:43.944686+0000","last_clean":"2024-09-16T02:46:35.253185+0000","last_became_active":"2024-09-16T02:47:26.504510+0000","last_became_peered":"2024-09-16T02:47:26.504510+0000","last_unstale":"2024-09-16T02:49:43.944686+0000","last_undegraded":"2024-09-16T02:49:43.944686+0000","last_fullsized":"2024-09-16T02:47:26.334503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":438,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3735,"num_bytes_recovered":3735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3735},{"shards":"2,3","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4045,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798367+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:45.798367+0000","last_peered":"2024-09-16T02:49:45.798367+0000","last_clean":"2024-09-16T02:49:45.798367+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:45.798367+0000","last_undegraded":"2024-09-16T02:49:45.798367+0000","last_fullsized":"2024-09-16T02:49:45.798367+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.784900+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:48:54.784900+0000","last_peered":"2024-09-16T02:48:54.784900+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:48:54.784900+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4025,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778960+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:48:54.778960+0000","last_peered":"2024-09-16T02:48:54.778960+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:48:54.778960+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4082,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778139+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:48:54.778139+0000","last_peered":"2024-09-16T02:48:54.778139+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:48:54.778139+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4050,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.779035+0000","last_change":"2024-09-16T02:47:26.521997+0000","last_active":"2024-09-16T02:48:54.779035+0000","last_peered":"2024-09-16T02:48:54.779035+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:48:54.779035+0000","last_undegraded":"2024-09-16T02:47:26.504474+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3997,"num_objects_misplaced":3997,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3975,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798253+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:45.798253+0000","last_peered":"2024-09-16T02:49:45.798253+0000","last_clean":"2024-09-16T02:49:45.798253+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:45.798253+0000","last_undegraded":"2024-09-16T02:49:45.798253+0000","last_fullsized":"2024-09-16T02:49:45.798253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:48:54.778635+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:48:54.778635+0000","last_peered":"2024-09-16T02:48:54.778635+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:48:54.778635+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20074,"num_objects_misplaced":20512,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10760,"num_bytes_recovered":76277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":325459968,"data_stored":642729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640510,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139672,"kb_used_data":46024,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94232168,"statfs":{"total":96636764160,"available":96493740032,"internally_reserved":0,"allocated":47128576,"data_stored":167019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673215,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127768,"kb_used_data":48008,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94244072,"statfs":{"total":96636764160,"available":96505929728,"internally_reserved":0,"allocated":49160192,"data_stored":170311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986203,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48902144,"data_stored":77442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46985216,"data_stored":76956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:49.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:50.234 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:50.234 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:50.255 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-16T02:49:49.700994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":257220,"kb_used_data":96292,"kb_used_omap":3,"kb_used_meta":160892,"kb_avail":188486460,"statfs":{"total":193273528320,"available":193010135040,"internally_reserved":0,"allocated":98603008,"data_stored":343448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":164754314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":1791,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002510"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4045,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798367+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:45.798367+0000","last_peered":"2024-09-16T02:49:45.798367+0000","last_clean":"2024-09-16T02:49:45.798367+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:45.798367+0000","last_undegraded":"2024-09-16T02:49:45.798367+0000","last_fullsized":"2024-09-16T02:49:45.798367+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4598,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:48.937705+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:48.937705+0000","last_peered":"2024-09-16T02:49:48.937705+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:48.937705+0000","last_undegraded":"2024-09-16T02:49:48.937705+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7632,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":181},{"shards":"2,3","objects":3816}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3975,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798253+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:45.798253+0000","last_peered":"2024-09-16T02:49:45.798253+0000","last_clean":"2024-09-16T02:49:45.798253+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:45.798253+0000","last_undegraded":"2024-09-16T02:49:45.798253+0000","last_fullsized":"2024-09-16T02:49:45.798253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327622656,"data_stored":643275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640511,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129452,"kb_used_data":48284,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94242388,"statfs":{"total":96636764160,"available":96504205312,"internally_reserved":0,"allocated":49442816,"data_stored":173137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673215,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127768,"kb_used_data":48008,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94244072,"statfs":{"total":96636764160,"available":96505929728,"internally_reserved":0,"allocated":49160192,"data_stored":170311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986203,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48902144,"data_stored":77442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49147904,"data_stored":77502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:50.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:50.630 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:50.651 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-16T02:49:49.700994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":257220,"kb_used_data":96292,"kb_used_omap":3,"kb_used_meta":160892,"kb_avail":188486460,"statfs":{"total":193273528320,"available":193010135040,"internally_reserved":0,"allocated":98603008,"data_stored":343448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":164754314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":1791,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002510"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4045,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798367+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:45.798367+0000","last_peered":"2024-09-16T02:49:45.798367+0000","last_clean":"2024-09-16T02:49:45.798367+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:45.798367+0000","last_undegraded":"2024-09-16T02:49:45.798367+0000","last_fullsized":"2024-09-16T02:49:45.798367+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4598,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:48.937705+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:48.937705+0000","last_peered":"2024-09-16T02:49:48.937705+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:48.937705+0000","last_undegraded":"2024-09-16T02:49:48.937705+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7632,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":181},{"shards":"2,3","objects":3816}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3975,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798253+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:45.798253+0000","last_peered":"2024-09-16T02:49:45.798253+0000","last_clean":"2024-09-16T02:49:45.798253+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:45.798253+0000","last_undegraded":"2024-09-16T02:49:45.798253+0000","last_fullsized":"2024-09-16T02:49:45.798253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327622656,"data_stored":643275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640511,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129452,"kb_used_data":48284,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94242388,"statfs":{"total":96636764160,"available":96504205312,"internally_reserved":0,"allocated":49442816,"data_stored":173137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673215,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127768,"kb_used_data":48008,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94244072,"statfs":{"total":96636764160,"available":96505929728,"internally_reserved":0,"allocated":49160192,"data_stored":170311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986203,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48902144,"data_stored":77442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49147904,"data_stored":77502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:50.651 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4598, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:48.937705+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:49:48.937705+0000', 'last_peered': '2024-09-16T02:49:48.937705+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:49:48.937705+0000', 'last_undegraded': '2024-09-16T02:49:48.937705+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7632, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 182, 'num_bytes_recovered': 218, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 181}, {'shards': '2,3', 'objects': 3816}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:50.652 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:50.653 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:50.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:51.104 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:51.104 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1250,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":263393280,"bytes_avail":193010135040,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":23709,"misplaced_total":63846,"misplaced_ratio":0.37134667794380227,"recovering_objects_per_sec":137,"recovering_bytes_per_sec":140,"recovering_keys_per_sec":0,"num_objects_recovered":1103,"num_bytes_recovered":1121,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:51.125 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:51.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:51.499 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:51.518 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-16T02:49:49.700994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":257220,"kb_used_data":96292,"kb_used_omap":3,"kb_used_meta":160892,"kb_avail":188486460,"statfs":{"total":193273528320,"available":193010135040,"internally_reserved":0,"allocated":98603008,"data_stored":343448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":164754314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":1791,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002510"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4045,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798367+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:45.798367+0000","last_peered":"2024-09-16T02:49:45.798367+0000","last_clean":"2024-09-16T02:49:45.798367+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:45.798367+0000","last_undegraded":"2024-09-16T02:49:45.798367+0000","last_fullsized":"2024-09-16T02:49:45.798367+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4598,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:48.937705+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:48.937705+0000","last_peered":"2024-09-16T02:49:48.937705+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:48.937705+0000","last_undegraded":"2024-09-16T02:49:48.937705+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7632,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":181},{"shards":"2,3","objects":3816}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3975,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-16T02:49:45.798253+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:45.798253+0000","last_peered":"2024-09-16T02:49:45.798253+0000","last_clean":"2024-09-16T02:49:45.798253+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:45.798253+0000","last_undegraded":"2024-09-16T02:49:45.798253+0000","last_fullsized":"2024-09-16T02:49:45.798253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327622656,"data_stored":643275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640511,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129452,"kb_used_data":48284,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94242388,"statfs":{"total":96636764160,"available":96504205312,"internally_reserved":0,"allocated":49442816,"data_stored":173137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673215,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127768,"kb_used_data":48008,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":94244072,"statfs":{"total":96636764160,"available":96505929728,"internally_reserved":0,"allocated":49160192,"data_stored":170311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986203,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48902144,"data_stored":77442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49147904,"data_stored":77502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98586624,"data_stored":207468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:54.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:54.870 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:54.890 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":656,"stamp":"2024-09-16T02:49:53.702066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":242496,"kb_used_data":98208,"kb_used_omap":3,"kb_used_meta":144252,"kb_avail":188501184,"statfs":{"total":193273528320,"available":193025212416,"internally_reserved":0,"allocated":100564992,"data_stored":346684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":147714954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":3197,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002382"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4598,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:48.937705+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:48.937705+0000","last_peered":"2024-09-16T02:49:48.937705+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:48.937705+0000","last_undegraded":"2024-09-16T02:49:48.937705+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7632,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":181},{"shards":"2,3","objects":3816}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313364480,"data_stored":639794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640512,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130384,"kb_used_data":49216,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94241456,"statfs":{"total":96636764160,"available":96503250944,"internally_reserved":0,"allocated":50397184,"data_stored":173370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112112,"kb_used_data":48992,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94259728,"statfs":{"total":96636764160,"available":96521961472,"internally_reserved":0,"allocated":50167808,"data_stored":173314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986204,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49897472,"data_stored":77685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50089984,"data_stored":77732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:54.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:55.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:55.212 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:55.232 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":656,"stamp":"2024-09-16T02:49:53.702066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":242496,"kb_used_data":98208,"kb_used_omap":3,"kb_used_meta":144252,"kb_avail":188501184,"statfs":{"total":193273528320,"available":193025212416,"internally_reserved":0,"allocated":100564992,"data_stored":346684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":147714954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":3197,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002382"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4598,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:48.937705+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:48.937705+0000","last_peered":"2024-09-16T02:49:48.937705+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:48.937705+0000","last_undegraded":"2024-09-16T02:49:48.937705+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7632,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":181},{"shards":"2,3","objects":3816}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23709,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11160,"num_bytes_recovered":76695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313364480,"data_stored":639794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640512,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130384,"kb_used_data":49216,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94241456,"statfs":{"total":96636764160,"available":96503250944,"internally_reserved":0,"allocated":50397184,"data_stored":173370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112112,"kb_used_data":48992,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94259728,"statfs":{"total":96636764160,"available":96521961472,"internally_reserved":0,"allocated":50167808,"data_stored":173314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986204,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49897472,"data_stored":77685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50089984,"data_stored":77732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:55.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:55.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:55.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:55.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:55.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:55.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:55.234 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:55.234 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 4598, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:48.937705+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:49:48.937705+0000', 'last_peered': '2024-09-16T02:49:48.937705+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:49:48.937705+0000', 'last_undegraded': '2024-09-16T02:49:48.937705+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7632, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 182, 'num_bytes_recovered': 218, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 181}, {'shards': '2,3', 'objects': 3816}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:55.234 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:55.234 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:55.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:49:55.657 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:55.657 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1255,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":248315904,"bytes_avail":193025212416,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":23709,"misplaced_total":63846,"misplaced_ratio":0.37134667794380227,"recovering_objects_per_sec":49,"recovering_bytes_per_sec":52,"recovering_keys_per_sec":0,"num_objects_recovered":400,"num_bytes_recovered":418,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:49:55.677 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:49:55.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:56.025 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:56.025 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:56.049 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-09-16T02:49:55.702909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23245,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11392,"num_bytes_recovered":76927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":242496,"kb_used_data":98208,"kb_used_omap":3,"kb_used_meta":144252,"kb_avail":188501184,"statfs":{"total":193273528320,"available":193025212416,"internally_reserved":0,"allocated":100564992,"data_stored":346684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":147714954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":2733,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.915684"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":5294,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:53.957919+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:53.957919+0000","last_peered":"2024-09-16T02:49:53.957919+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:53.957919+0000","last_undegraded":"2024-09-16T02:49:53.957919+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7168,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":413},{"shards":"2,3","objects":3584}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23245,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11392,"num_bytes_recovered":76927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313364480,"data_stored":639794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640512,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130384,"kb_used_data":49216,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94241456,"statfs":{"total":96636764160,"available":96503250944,"internally_reserved":0,"allocated":50397184,"data_stored":173370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112112,"kb_used_data":48992,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94259728,"statfs":{"total":96636764160,"available":96521961472,"internally_reserved":0,"allocated":50167808,"data_stored":173314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986204,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49897472,"data_stored":77685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50089984,"data_stored":77732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:59.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:59.402 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:59.402 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:59.420 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":658,"stamp":"2024-09-16T02:49:57.703336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23245,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11392,"num_bytes_recovered":76927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":248588,"kb_used_data":100140,"kb_used_omap":3,"kb_used_meta":148412,"kb_avail":188495092,"statfs":{"total":193273528320,"available":193018974208,"internally_reserved":0,"allocated":102543360,"data_stored":347167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":151974794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":2733,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":632,"num_bytes_recovered":650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.914351"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":5294,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:53.957919+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:53.957919+0000","last_peered":"2024-09-16T02:49:53.957919+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:53.957919+0000","last_undegraded":"2024-09-16T02:49:53.957919+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7168,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":413},{"shards":"2,3","objects":3584}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":23245,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11392,"num_bytes_recovered":76927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315338752,"data_stored":640276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135488,"kb_used_data":50160,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94236352,"statfs":{"total":96636764160,"available":96498024448,"internally_reserved":0,"allocated":51363840,"data_stored":173606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673217,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113100,"kb_used_data":49980,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94258740,"statfs":{"total":96636764160,"available":96520949760,"internally_reserved":0,"allocated":51179520,"data_stored":173561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986205,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50896896,"data_stored":77929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51064832,"data_stored":77970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:59.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:49:59.752 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:49:59.752 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:49:59.773 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-09-16T02:49:59.703917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":248588,"kb_used_data":100140,"kb_used_omap":3,"kb_used_meta":148412,"kb_avail":188495092,"statfs":{"total":193273528320,"available":193018974208,"internally_reserved":0,"allocated":102543360,"data_stored":347167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":151974794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":2689,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003556"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6017,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:58.967186+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:58.967186+0000","last_peered":"2024-09-16T02:49:58.967186+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:58.967186+0000","last_undegraded":"2024-09-16T02:49:58.967186+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6686,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":655,"num_bytes_recovered":691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":654},{"shards":"2,3","objects":3343}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315338752,"data_stored":640276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135488,"kb_used_data":50160,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94236352,"statfs":{"total":96636764160,"available":96498024448,"internally_reserved":0,"allocated":51363840,"data_stored":173606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673217,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113100,"kb_used_data":49980,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94258740,"statfs":{"total":96636764160,"available":96520949760,"internally_reserved":0,"allocated":51179520,"data_stored":173561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986205,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50896896,"data_stored":77929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51064832,"data_stored":77970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:49:59.774 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:49:59.774 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:59.774 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:49:59.774 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:59.774 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:49:59.775 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:59.775 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:49:59.775 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 6017, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:58.967186+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:49:58.967186+0000', 'last_peered': '2024-09-16T02:49:58.967186+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:49:58.967186+0000', 'last_undegraded': '2024-09-16T02:49:58.967186+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6686, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 655, 'num_bytes_recovered': 691, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 654}, {'shards': '2,3', 'objects': 3343}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:59.775 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:49:59.775 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:49:59.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:00.180 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:00.180 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1259,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":254554112,"bytes_avail":193018974208,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":22763,"misplaced_total":63846,"misplaced_ratio":0.3565297747705416,"recovering_objects_per_sec":54,"recovering_bytes_per_sec":55,"recovering_keys_per_sec":0,"num_objects_recovered":654,"num_bytes_recovered":672,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:00.206 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:00.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:00.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:00.529 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:00.547 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-09-16T02:49:59.703917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":248588,"kb_used_data":100140,"kb_used_omap":3,"kb_used_meta":148412,"kb_avail":188495092,"statfs":{"total":193273528320,"available":193018974208,"internally_reserved":0,"allocated":102543360,"data_stored":347167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":151974794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3997,"num_objects_misplaced":2689,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":654,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003556"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6017,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:58.967186+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:58.967186+0000","last_peered":"2024-09-16T02:49:58.967186+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:58.967186+0000","last_undegraded":"2024-09-16T02:49:58.967186+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6686,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":655,"num_bytes_recovered":691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":654},{"shards":"2,3","objects":3343}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315338752,"data_stored":640276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640513,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":135488,"kb_used_data":50160,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94236352,"statfs":{"total":96636764160,"available":96498024448,"internally_reserved":0,"allocated":51363840,"data_stored":173606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673217,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113100,"kb_used_data":49980,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94258740,"statfs":{"total":96636764160,"available":96520949760,"internally_reserved":0,"allocated":51179520,"data_stored":173561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986205,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50896896,"data_stored":77929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51064832,"data_stored":77970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:03.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:03.916 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:03.917 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:03.939 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2024-09-16T02:50:03.705166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":250504,"kb_used_data":102056,"kb_used_omap":3,"kb_used_meta":148412,"kb_avail":188493176,"statfs":{"total":193273528320,"available":193017012224,"internally_reserved":0,"allocated":104505344,"data_stored":347646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":151974794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-946,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6017,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:58.967186+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:58.967186+0000","last_peered":"2024-09-16T02:49:58.967186+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:58.967186+0000","last_undegraded":"2024-09-16T02:49:58.967186+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6686,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":655,"num_bytes_recovered":691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":654},{"shards":"2,3","objects":3343}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317444096,"data_stored":640790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136380,"kb_used_data":51052,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94235460,"statfs":{"total":96636764160,"available":96497111040,"internally_reserved":0,"allocated":52277248,"data_stored":173829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673218,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114124,"kb_used_data":51004,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94257716,"statfs":{"total":96636764160,"available":96519901184,"internally_reserved":0,"allocated":52228096,"data_stored":173817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986206,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51945472,"data_stored":78185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52121600,"data_stored":78228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:03.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:04.270 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:04.270 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:04.290 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2024-09-16T02:50:03.705166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":250504,"kb_used_data":102056,"kb_used_omap":3,"kb_used_meta":148412,"kb_avail":188493176,"statfs":{"total":193273528320,"available":193017012224,"internally_reserved":0,"allocated":104505344,"data_stored":347646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":151974794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-946,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6017,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:58.967186+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:58.967186+0000","last_peered":"2024-09-16T02:49:58.967186+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:58.967186+0000","last_undegraded":"2024-09-16T02:49:58.967186+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6686,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":655,"num_bytes_recovered":691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":654},{"shards":"2,3","objects":3343}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317444096,"data_stored":640790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136380,"kb_used_data":51052,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94235460,"statfs":{"total":96636764160,"available":96497111040,"internally_reserved":0,"allocated":52277248,"data_stored":173829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673218,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114124,"kb_used_data":51004,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94257716,"statfs":{"total":96636764160,"available":96519901184,"internally_reserved":0,"allocated":52228096,"data_stored":173817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986206,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51945472,"data_stored":78185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52121600,"data_stored":78228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:04.291 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:04.291 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:04.292 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:04.292 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:04.292 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:04.292 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:04.293 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:04.293 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 6017, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:49:58.967186+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:49:58.967186+0000', 'last_peered': '2024-09-16T02:49:58.967186+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:49:58.967186+0000', 'last_undegraded': '2024-09-16T02:49:58.967186+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6686, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 655, 'num_bytes_recovered': 691, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 654}, {'shards': '2,3', 'objects': 3343}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:04.293 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:04.293 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:04.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:04.699 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:04.699 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1264,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":256516096,"bytes_avail":193017012224,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":22763,"misplaced_total":63846,"misplaced_ratio":0.3565297747705416,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":39,"recovering_keys_per_sec":0,"num_objects_recovered":473,"num_bytes_recovered":473,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:04.721 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:04.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:05.041 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:05.041 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:05.060 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2024-09-16T02:50:03.705166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":250504,"kb_used_data":102056,"kb_used_omap":3,"kb_used_meta":148412,"kb_avail":188493176,"statfs":{"total":193273528320,"available":193017012224,"internally_reserved":0,"allocated":104505344,"data_stored":347646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":151974794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-946,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6017,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:49:58.967186+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:49:58.967186+0000","last_peered":"2024-09-16T02:49:58.967186+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:49:58.967186+0000","last_undegraded":"2024-09-16T02:49:58.967186+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6686,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":655,"num_bytes_recovered":691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":654},{"shards":"2,3","objects":3343}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22763,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11633,"num_bytes_recovered":77168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317444096,"data_stored":640790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136380,"kb_used_data":51052,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94235460,"statfs":{"total":96636764160,"available":96497111040,"internally_reserved":0,"allocated":52277248,"data_stored":173829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673218,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114124,"kb_used_data":51004,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94257716,"statfs":{"total":96636764160,"available":96519901184,"internally_reserved":0,"allocated":52228096,"data_stored":173817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986206,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51945472,"data_stored":78185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52121600,"data_stored":78228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:08.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:08.398 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:08.398 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:08.417 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-09-16T02:50:07.706518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22223,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11903,"num_bytes_recovered":77438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":256872,"kb_used_data":104264,"kb_used_omap":3,"kb_used_meta":152572,"kb_avail":188486808,"statfs":{"total":193273528320,"available":193010491392,"internally_reserved":0,"allocated":106766336,"data_stored":348198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":156234634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1022,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003608"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6827,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:03.981698+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:03.981698+0000","last_peered":"2024-09-16T02:50:03.981698+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:03.981698+0000","last_undegraded":"2024-09-16T02:50:03.981698+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6146,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":925,"num_bytes_recovered":961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":924},{"shards":"2,3","objects":3073}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22223,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11903,"num_bytes_recovered":77438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":319684608,"data_stored":641337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137496,"kb_used_data":52168,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94234344,"statfs":{"total":96636764160,"available":96495968256,"internally_reserved":0,"allocated":53420032,"data_stored":174108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673219,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119376,"kb_used_data":52096,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94252464,"statfs":{"total":96636764160,"available":96514523136,"internally_reserved":0,"allocated":53346304,"data_stored":174090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986207,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53059584,"data_stored":78457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248000,"data_stored":78503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:08.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:08.735 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:08.736 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:08.756 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-09-16T02:50:07.706518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22223,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11903,"num_bytes_recovered":77438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":256872,"kb_used_data":104264,"kb_used_omap":3,"kb_used_meta":152572,"kb_avail":188486808,"statfs":{"total":193273528320,"available":193010491392,"internally_reserved":0,"allocated":106766336,"data_stored":348198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":156234634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1022,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003608"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6827,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:03.981698+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:03.981698+0000","last_peered":"2024-09-16T02:50:03.981698+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:03.981698+0000","last_undegraded":"2024-09-16T02:50:03.981698+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6146,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":925,"num_bytes_recovered":961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":924},{"shards":"2,3","objects":3073}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22223,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11903,"num_bytes_recovered":77438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":319684608,"data_stored":641337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137496,"kb_used_data":52168,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94234344,"statfs":{"total":96636764160,"available":96495968256,"internally_reserved":0,"allocated":53420032,"data_stored":174108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673219,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119376,"kb_used_data":52096,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94252464,"statfs":{"total":96636764160,"available":96514523136,"internally_reserved":0,"allocated":53346304,"data_stored":174090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986207,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53059584,"data_stored":78457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248000,"data_stored":78503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:08.757 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 6827, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:03.981698+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:03.981698+0000', 'last_peered': '2024-09-16T02:50:03.981698+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:03.981698+0000', 'last_undegraded': '2024-09-16T02:50:03.981698+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6146, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 925, 'num_bytes_recovered': 961, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 924}, {'shards': '2,3', 'objects': 3073}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:08.758 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:08.758 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:08.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:09.194 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:09.194 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1268,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":263036928,"bytes_avail":193010491392,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":22223,"misplaced_total":63846,"misplaced_ratio":0.34807192306487489,"recovering_objects_per_sec":42,"recovering_bytes_per_sec":42,"recovering_keys_per_sec":0,"num_objects_recovered":511,"num_bytes_recovered":511,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:09.214 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:09.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:09.525 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:09.526 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:09.544 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-09-16T02:50:07.706518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22223,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11903,"num_bytes_recovered":77438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":256872,"kb_used_data":104264,"kb_used_omap":3,"kb_used_meta":152572,"kb_avail":188486808,"statfs":{"total":193273528320,"available":193010491392,"internally_reserved":0,"allocated":106766336,"data_stored":348198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":156234634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1022,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003608"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":6827,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:03.981698+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:03.981698+0000","last_peered":"2024-09-16T02:50:03.981698+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:03.981698+0000","last_undegraded":"2024-09-16T02:50:03.981698+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6146,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":925,"num_bytes_recovered":961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":924},{"shards":"2,3","objects":3073}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":22223,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11903,"num_bytes_recovered":77438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":319684608,"data_stored":641337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137496,"kb_used_data":52168,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94234344,"statfs":{"total":96636764160,"available":96495968256,"internally_reserved":0,"allocated":53420032,"data_stored":174108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673219,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119376,"kb_used_data":52096,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94252464,"statfs":{"total":96636764160,"available":96514523136,"internally_reserved":0,"allocated":53346304,"data_stored":174090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986207,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53059584,"data_stored":78457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248000,"data_stored":78503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:12.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:12.880 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:12.903 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2024-09-16T02:50:11.707591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":21607,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12211,"num_bytes_recovered":77746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":257976,"kb_used_data":105368,"kb_used_omap":3,"kb_used_meta":152572,"kb_avail":188485704,"statfs":{"total":193273528320,"available":193009360896,"internally_reserved":0,"allocated":107896832,"data_stored":348474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":156234634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1156,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":7751,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:08.986788+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:08.986788+0000","last_peered":"2024-09-16T02:50:08.986788+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:08.986788+0000","last_undegraded":"2024-09-16T02:50:08.986788+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5530,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":1269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1232},{"shards":"2,3","objects":2765}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":21607,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12211,"num_bytes_recovered":77746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":320823296,"data_stored":641615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137496,"kb_used_data":52168,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94234344,"statfs":{"total":96636764160,"available":96495968256,"internally_reserved":0,"allocated":53420032,"data_stored":174108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120480,"kb_used_data":53200,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94251360,"statfs":{"total":96636764160,"available":96513392640,"internally_reserved":0,"allocated":54476800,"data_stored":174366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986208,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54198272,"data_stored":78735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248000,"data_stored":78503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:12.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:13.225 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:13.225 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:13.245 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2024-09-16T02:50:11.707591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":21607,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12211,"num_bytes_recovered":77746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":257976,"kb_used_data":105368,"kb_used_omap":3,"kb_used_meta":152572,"kb_avail":188485704,"statfs":{"total":193273528320,"available":193009360896,"internally_reserved":0,"allocated":107896832,"data_stored":348474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":156234634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1156,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":7751,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:08.986788+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:08.986788+0000","last_peered":"2024-09-16T02:50:08.986788+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:08.986788+0000","last_undegraded":"2024-09-16T02:50:08.986788+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5530,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":1269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1232},{"shards":"2,3","objects":2765}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":21607,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12211,"num_bytes_recovered":77746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":320823296,"data_stored":641615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137496,"kb_used_data":52168,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94234344,"statfs":{"total":96636764160,"available":96495968256,"internally_reserved":0,"allocated":53420032,"data_stored":174108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120480,"kb_used_data":53200,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94251360,"statfs":{"total":96636764160,"available":96513392640,"internally_reserved":0,"allocated":54476800,"data_stored":174366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986208,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54198272,"data_stored":78735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248000,"data_stored":78503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:13.246 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:13.246 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:13.246 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:13.246 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:13.247 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:13.247 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:13.247 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:13.247 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 7751, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:08.986788+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:08.986788+0000', 'last_peered': '2024-09-16T02:50:08.986788+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:08.986788+0000', 'last_undegraded': '2024-09-16T02:50:08.986788+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5530, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1233, 'num_bytes_recovered': 1269, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1232}, {'shards': '2,3', 'objects': 2765}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:13.247 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:13.247 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:13.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:13.641 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:13.641 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1273,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":264167424,"bytes_avail":193009360896,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":21607,"misplaced_total":63846,"misplaced_ratio":0.33842370704507724,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":48,"recovering_keys_per_sec":0,"num_objects_recovered":578,"num_bytes_recovered":578,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:13.663 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:13.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:14.043 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:14.043 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:14.064 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2024-09-16T02:50:13.708168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":21607,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12211,"num_bytes_recovered":77746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":259092,"kb_used_data":106484,"kb_used_omap":3,"kb_used_meta":152572,"kb_avail":188484588,"statfs":{"total":193273528320,"available":193008218112,"internally_reserved":0,"allocated":109039616,"data_stored":348753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":156234634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1156,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003705"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":7751,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:08.986788+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:08.986788+0000","last_peered":"2024-09-16T02:50:08.986788+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:08.986788+0000","last_undegraded":"2024-09-16T02:50:08.986788+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5530,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":1269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1232},{"shards":"2,3","objects":2765}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":21607,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12211,"num_bytes_recovered":77746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":321953792,"data_stored":641891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640516,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138612,"kb_used_data":53284,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94233228,"statfs":{"total":96636764160,"available":96494825472,"internally_reserved":0,"allocated":54562816,"data_stored":174387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120480,"kb_used_data":53200,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94251360,"statfs":{"total":96636764160,"available":96513392640,"internally_reserved":0,"allocated":54476800,"data_stored":174366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986208,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54198272,"data_stored":78735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54378496,"data_stored":78779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:17.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:17.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:17.426 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:17.447 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-09-16T02:50:15.708552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20985,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12522,"num_bytes_recovered":78057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":259092,"kb_used_data":106484,"kb_used_omap":3,"kb_used_meta":152572,"kb_avail":188484588,"statfs":{"total":193273528320,"available":193008218112,"internally_reserved":0,"allocated":109039616,"data_stored":348753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":156234634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1778,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":889,"num_bytes_recovered":889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003386"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":8684,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:13.993489+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:13.993489+0000","last_peered":"2024-09-16T02:50:13.993489+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:13.993489+0000","last_undegraded":"2024-09-16T02:50:13.993489+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4908,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1543},{"shards":"2,3","objects":2454}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20985,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12522,"num_bytes_recovered":78057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":321953792,"data_stored":641891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640516,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138612,"kb_used_data":53284,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94233228,"statfs":{"total":96636764160,"available":96494825472,"internally_reserved":0,"allocated":54562816,"data_stored":174387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120480,"kb_used_data":53200,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94251360,"statfs":{"total":96636764160,"available":96513392640,"internally_reserved":0,"allocated":54476800,"data_stored":174366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986208,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54198272,"data_stored":78735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54378496,"data_stored":78779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:17.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:17.800 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:17.820 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-09-16T02:50:17.709286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20985,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12522,"num_bytes_recovered":78057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":301800,"kb_used_data":108936,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188441880,"statfs":{"total":193273528320,"available":192964485120,"internally_reserved":0,"allocated":111550464,"data_stored":349366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1238,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003653"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":8684,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:13.993489+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:13.993489+0000","last_peered":"2024-09-16T02:50:13.993489+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:13.993489+0000","last_undegraded":"2024-09-16T02:50:13.993489+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4908,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1543},{"shards":"2,3","objects":2454}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20985,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12522,"num_bytes_recovered":78057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":324534272,"data_stored":642521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640517,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162040,"kb_used_data":54504,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94209800,"statfs":{"total":96636764160,"available":96470835200,"internally_reserved":0,"allocated":55812096,"data_stored":174692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673221,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139760,"kb_used_data":54432,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94232080,"statfs":{"total":96636764160,"available":96493649920,"internally_reserved":0,"allocated":55738368,"data_stored":174674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986209,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55447552,"data_stored":79040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55709696,"data_stored":79104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 8684, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:13.993489+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:13.993489+0000', 'last_peered': '2024-09-16T02:50:13.993489+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:13.993489+0000', 'last_undegraded': '2024-09-16T02:50:13.993489+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4908, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1544, 'num_bytes_recovered': 1580, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1543}, {'shards': '2,3', 'objects': 2454}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:17.821 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:17.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:18.225 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:18.225 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1277,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":309043200,"bytes_avail":192964485120,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":20985,"misplaced_total":63846,"misplaced_ratio":0.32868151489521663,"recovering_objects_per_sec":51,"recovering_bytes_per_sec":51,"recovering_keys_per_sec":0,"num_objects_recovered":619,"num_bytes_recovered":619,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:18.246 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:18.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:18.586 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:18.586 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:18.607 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-09-16T02:50:17.709286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20985,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12522,"num_bytes_recovered":78057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":301800,"kb_used_data":108936,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188441880,"statfs":{"total":193273528320,"available":192964485120,"internally_reserved":0,"allocated":111550464,"data_stored":349366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1238,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003653"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":8684,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:13.993489+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:13.993489+0000","last_peered":"2024-09-16T02:50:13.993489+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:13.993489+0000","last_undegraded":"2024-09-16T02:50:13.993489+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4908,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1543},{"shards":"2,3","objects":2454}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20985,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12522,"num_bytes_recovered":78057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":324534272,"data_stored":642521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640517,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162040,"kb_used_data":54504,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94209800,"statfs":{"total":96636764160,"available":96470835200,"internally_reserved":0,"allocated":55812096,"data_stored":174692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673221,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139760,"kb_used_data":54432,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94232080,"statfs":{"total":96636764160,"available":96493649920,"internally_reserved":0,"allocated":55738368,"data_stored":174674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986209,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55447552,"data_stored":79040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55709696,"data_stored":79104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:21.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:21.959 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:21.960 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:21.982 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2024-09-16T02:50:21.710461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20321,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12854,"num_bytes_recovered":78389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":303144,"kb_used_data":110280,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188440536,"statfs":{"total":193273528320,"available":192963108864,"internally_reserved":0,"allocated":112926720,"data_stored":349702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1286,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":643,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":9680,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:18.996600+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:18.996600+0000","last_peered":"2024-09-16T02:50:18.996600+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:18.996600+0000","last_undegraded":"2024-09-16T02:50:18.996600+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4244,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":1912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1875},{"shards":"2,3","objects":2122}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20321,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12854,"num_bytes_recovered":78389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":325902336,"data_stored":642855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640517,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162040,"kb_used_data":54504,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94209800,"statfs":{"total":96636764160,"available":96470835200,"internally_reserved":0,"allocated":55812096,"data_stored":174692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673222,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141104,"kb_used_data":55776,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94230736,"statfs":{"total":96636764160,"available":96492273664,"internally_reserved":0,"allocated":57114624,"data_stored":175010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986210,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56815616,"data_stored":79374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55709696,"data_stored":79104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:21.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:22.312 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:22.312 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:22.332 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2024-09-16T02:50:21.710461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20321,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12854,"num_bytes_recovered":78389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":303144,"kb_used_data":110280,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188440536,"statfs":{"total":193273528320,"available":192963108864,"internally_reserved":0,"allocated":112926720,"data_stored":349702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1286,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":643,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":9680,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:18.996600+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:18.996600+0000","last_peered":"2024-09-16T02:50:18.996600+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:18.996600+0000","last_undegraded":"2024-09-16T02:50:18.996600+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4244,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":1912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1875},{"shards":"2,3","objects":2122}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20321,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12854,"num_bytes_recovered":78389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":325902336,"data_stored":642855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640517,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162040,"kb_used_data":54504,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94209800,"statfs":{"total":96636764160,"available":96470835200,"internally_reserved":0,"allocated":55812096,"data_stored":174692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673222,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141104,"kb_used_data":55776,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94230736,"statfs":{"total":96636764160,"available":96492273664,"internally_reserved":0,"allocated":57114624,"data_stored":175010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986210,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56815616,"data_stored":79374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55709696,"data_stored":79104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:22.333 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:22.333 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:22.333 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:22.333 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:22.333 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:22.334 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:22.334 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:22.334 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 9680, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:18.996600+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:18.996600+0000', 'last_peered': '2024-09-16T02:50:18.996600+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:18.996600+0000', 'last_undegraded': '2024-09-16T02:50:18.996600+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4244, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1876, 'num_bytes_recovered': 1912, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1875}, {'shards': '2,3', 'objects': 2122}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:22.334 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:22.334 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:22.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:22.734 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1282,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":310419456,"bytes_avail":192963108864,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":20321,"misplaced_total":63846,"misplaced_ratio":0.31828148983491528,"recovering_objects_per_sec":53,"recovering_bytes_per_sec":53,"recovering_keys_per_sec":0,"num_objects_recovered":643,"num_bytes_recovered":643,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:22.754 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:22.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:23.073 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:23.092 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2024-09-16T02:50:21.710461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20321,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12854,"num_bytes_recovered":78389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":303144,"kb_used_data":110280,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188440536,"statfs":{"total":193273528320,"available":192963108864,"internally_reserved":0,"allocated":112926720,"data_stored":349702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1286,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":643,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":9680,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:18.996600+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:18.996600+0000","last_peered":"2024-09-16T02:50:18.996600+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:18.996600+0000","last_undegraded":"2024-09-16T02:50:18.996600+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4244,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":1912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1875},{"shards":"2,3","objects":2122}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":20321,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12854,"num_bytes_recovered":78389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":325902336,"data_stored":642855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640517,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162040,"kb_used_data":54504,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94209800,"statfs":{"total":96636764160,"available":96470835200,"internally_reserved":0,"allocated":55812096,"data_stored":174692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673222,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141104,"kb_used_data":55776,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94230736,"statfs":{"total":96636764160,"available":96492273664,"internally_reserved":0,"allocated":57114624,"data_stored":175010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986210,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56815616,"data_stored":79374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55709696,"data_stored":79104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:26.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:26.442 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:26.443 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:26.465 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2024-09-16T02:50:25.711572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":19603,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13213,"num_bytes_recovered":78748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":304496,"kb_used_data":111632,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188439184,"statfs":{"total":193273528320,"available":192961724416,"internally_reserved":0,"allocated":114311168,"data_stored":350040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2004,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1002,"num_bytes_recovered":1002,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003404"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":10757,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:24.002676+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:24.002676+0000","last_peered":"2024-09-16T02:50:24.002676+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:24.002676+0000","last_undegraded":"2024-09-16T02:50:24.002676+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2235,"num_bytes_recovered":2271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2234},{"shards":"2,3","objects":1763}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":19603,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13213,"num_bytes_recovered":78748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327405568,"data_stored":643222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163392,"kb_used_data":55856,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94208448,"statfs":{"total":96636764160,"available":96469450752,"internally_reserved":0,"allocated":57196544,"data_stored":175030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673222,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141104,"kb_used_data":55776,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94230736,"statfs":{"total":96636764160,"available":96492273664,"internally_reserved":0,"allocated":57114624,"data_stored":175010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986210,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56815616,"data_stored":79374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57212928,"data_stored":79471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:26.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:26.792 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:26.792 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:26.812 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2024-09-16T02:50:25.711572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":19603,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13213,"num_bytes_recovered":78748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":304496,"kb_used_data":111632,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188439184,"statfs":{"total":193273528320,"available":192961724416,"internally_reserved":0,"allocated":114311168,"data_stored":350040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2004,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1002,"num_bytes_recovered":1002,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003404"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":10757,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:24.002676+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:24.002676+0000","last_peered":"2024-09-16T02:50:24.002676+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:24.002676+0000","last_undegraded":"2024-09-16T02:50:24.002676+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2235,"num_bytes_recovered":2271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2234},{"shards":"2,3","objects":1763}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":19603,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13213,"num_bytes_recovered":78748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327405568,"data_stored":643222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163392,"kb_used_data":55856,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94208448,"statfs":{"total":96636764160,"available":96469450752,"internally_reserved":0,"allocated":57196544,"data_stored":175030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673222,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141104,"kb_used_data":55776,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94230736,"statfs":{"total":96636764160,"available":96492273664,"internally_reserved":0,"allocated":57114624,"data_stored":175010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986210,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56815616,"data_stored":79374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57212928,"data_stored":79471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:26.813 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:26.813 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:26.813 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:26.813 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:26.813 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:26.813 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:26.813 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:26.814 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 10757, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:24.002676+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:24.002676+0000', 'last_peered': '2024-09-16T02:50:24.002676+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:24.002676+0000', 'last_undegraded': '2024-09-16T02:50:24.002676+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3526, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2235, 'num_bytes_recovered': 2271, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2234}, {'shards': '2,3', 'objects': 1763}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:26.814 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:26.814 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:26.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:27.200 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:27.200 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1286,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":311803904,"bytes_avail":192961724416,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":19603,"misplaced_total":63846,"misplaced_ratio":0.30703567960404726,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":83,"recovering_keys_per_sec":0,"num_objects_recovered":1002,"num_bytes_recovered":1002,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:27.222 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:27.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:27.565 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:27.565 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:27.583 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2024-09-16T02:50:25.711572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":19603,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13213,"num_bytes_recovered":78748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":304496,"kb_used_data":111632,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188439184,"statfs":{"total":193273528320,"available":192961724416,"internally_reserved":0,"allocated":114311168,"data_stored":350040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2004,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1002,"num_bytes_recovered":1002,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003404"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":10757,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:24.002676+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:24.002676+0000","last_peered":"2024-09-16T02:50:24.002676+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:24.002676+0000","last_undegraded":"2024-09-16T02:50:24.002676+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3526,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2235,"num_bytes_recovered":2271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2234},{"shards":"2,3","objects":1763}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":19603,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13213,"num_bytes_recovered":78748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327405568,"data_stored":643222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640518,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163392,"kb_used_data":55856,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94208448,"statfs":{"total":96636764160,"available":96469450752,"internally_reserved":0,"allocated":57196544,"data_stored":175030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673222,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141104,"kb_used_data":55776,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94230736,"statfs":{"total":96636764160,"available":96492273664,"internally_reserved":0,"allocated":57114624,"data_stored":175010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986210,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56815616,"data_stored":79374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57212928,"data_stored":79471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:30.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:30.939 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:30.961 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2024-09-16T02:50:29.712602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":311848,"kb_used_data":114824,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188431832,"statfs":{"total":193273528320,"available":192954195968,"internally_reserved":0,"allocated":117579776,"data_stored":350838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2186,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":1093,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003316"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":11963,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:29.011169+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:29.011169+0000","last_peered":"2024-09-16T02:50:29.011169+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:29.011169+0000","last_undegraded":"2024-09-16T02:50:29.011169+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2722,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2637,"num_bytes_recovered":2673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2636},{"shards":"2,3","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330747904,"data_stored":644038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165028,"kb_used_data":57492,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94206812,"statfs":{"total":96636764160,"available":96467775488,"internally_reserved":0,"allocated":58871808,"data_stored":175439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673223,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146820,"kb_used_data":57332,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94225020,"statfs":{"total":96636764160,"available":96486420480,"internally_reserved":0,"allocated":58707968,"data_stored":175399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986211,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58429440,"data_stored":79768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58941440,"data_stored":79893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:30.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:31.312 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:31.333 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2024-09-16T02:50:29.712602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":311848,"kb_used_data":114824,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188431832,"statfs":{"total":193273528320,"available":192954195968,"internally_reserved":0,"allocated":117579776,"data_stored":350838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2186,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":1093,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003316"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":11963,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:29.011169+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:29.011169+0000","last_peered":"2024-09-16T02:50:29.011169+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:29.011169+0000","last_undegraded":"2024-09-16T02:50:29.011169+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2722,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2637,"num_bytes_recovered":2673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2636},{"shards":"2,3","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330747904,"data_stored":644038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165028,"kb_used_data":57492,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94206812,"statfs":{"total":96636764160,"available":96467775488,"internally_reserved":0,"allocated":58871808,"data_stored":175439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673223,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146820,"kb_used_data":57332,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94225020,"statfs":{"total":96636764160,"available":96486420480,"internally_reserved":0,"allocated":58707968,"data_stored":175399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986211,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":58429440,"data_stored":79768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58941440,"data_stored":79893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:31.334 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:31.334 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:31.334 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:31.335 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:31.335 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:31.335 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:31.335 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:31.335 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 11963, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:29.011169+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:29.011169+0000', 'last_peered': '2024-09-16T02:50:29.011169+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:29.011169+0000', 'last_undegraded': '2024-09-16T02:50:29.011169+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2722, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2637, 'num_bytes_recovered': 2673, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2636}, {'shards': '2,3', 'objects': 1361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:31.335 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:31.335 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:31.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:31.752 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:31.752 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1291,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":319332352,"bytes_avail":192954195968,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":18799,"misplaced_total":63846,"misplaced_ratio":0.29444287817561005,"recovering_objects_per_sec":91,"recovering_bytes_per_sec":91,"recovering_keys_per_sec":0,"num_objects_recovered":1093,"num_bytes_recovered":1093,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:31.773 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:31.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:32.123 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:32.144 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":675,"stamp":"2024-09-16T02:50:31.713213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":313700,"kb_used_data":116676,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188429980,"statfs":{"total":193273528320,"available":192952299520,"internally_reserved":0,"allocated":119476224,"data_stored":351301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1522,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":11963,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:29.011169+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:29.011169+0000","last_peered":"2024-09-16T02:50:29.011169+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:29.011169+0000","last_undegraded":"2024-09-16T02:50:29.011169+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2722,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2637,"num_bytes_recovered":2673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2636},{"shards":"2,3","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":332685312,"data_stored":644511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165028,"kb_used_data":57492,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94206812,"statfs":{"total":96636764160,"available":96467775488,"internally_reserved":0,"allocated":58871808,"data_stored":175439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148672,"kb_used_data":59184,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94223168,"statfs":{"total":96636764160,"available":96484524032,"internally_reserved":0,"allocated":60604416,"data_stored":175862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986212,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60366848,"data_stored":80241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58941440,"data_stored":79893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:35.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:35.496 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:35.497 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:35.517 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":676,"stamp":"2024-09-16T02:50:33.713720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":319720,"kb_used_data":118536,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188423960,"statfs":{"total":193273528320,"available":192946135040,"internally_reserved":0,"allocated":121380864,"data_stored":351766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1522,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":11963,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:29.011169+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:29.011169+0000","last_peered":"2024-09-16T02:50:29.011169+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:29.011169+0000","last_undegraded":"2024-09-16T02:50:29.011169+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2722,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2637,"num_bytes_recovered":2673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2636},{"shards":"2,3","objects":1361}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":18799,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13615,"num_bytes_recovered":79150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":334667776,"data_stored":644995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171048,"kb_used_data":59352,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94200792,"statfs":{"total":96636764160,"available":96461611008,"internally_reserved":0,"allocated":60776448,"data_stored":175904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148672,"kb_used_data":59184,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94223168,"statfs":{"total":96636764160,"available":96484524032,"internally_reserved":0,"allocated":60604416,"data_stored":175862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986212,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60366848,"data_stored":80241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60923904,"data_stored":80377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:35.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:35.862 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:35.883 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2024-09-16T02:50:35.714495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":17805,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14112,"num_bytes_recovered":79647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":319720,"kb_used_data":118536,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188423960,"statfs":{"total":193273528320,"available":192946135040,"internally_reserved":0,"allocated":121380864,"data_stored":351766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2516,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":1258,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003576"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":13454,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:34.013586+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:34.013586+0000","last_peered":"2024-09-16T02:50:34.013586+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:34.013586+0000","last_undegraded":"2024-09-16T02:50:34.013586+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1728,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3134,"num_bytes_recovered":3170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3133},{"shards":"2,3","objects":864}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":17805,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14112,"num_bytes_recovered":79647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":334667776,"data_stored":644995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171048,"kb_used_data":59352,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94200792,"statfs":{"total":96636764160,"available":96461611008,"internally_reserved":0,"allocated":60776448,"data_stored":175904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148672,"kb_used_data":59184,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94223168,"statfs":{"total":96636764160,"available":96484524032,"internally_reserved":0,"allocated":60604416,"data_stored":175862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986212,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60366848,"data_stored":80241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60923904,"data_stored":80377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:35.884 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:35.884 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:35.885 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:35.885 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:35.885 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:35.885 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:35.885 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:35.886 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 13454, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:34.013586+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:34.013586+0000', 'last_peered': '2024-09-16T02:50:34.013586+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:34.013586+0000', 'last_undegraded': '2024-09-16T02:50:34.013586+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1728, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3134, 'num_bytes_recovered': 3170, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3133}, {'shards': '2,3', 'objects': 864}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:35.886 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:35.886 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:35.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:36.308 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:36.309 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1295,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":327393280,"bytes_avail":192946135040,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":17805,"misplaced_total":63846,"misplaced_ratio":0.27887416596184567,"recovering_objects_per_sec":104,"recovering_bytes_per_sec":104,"recovering_keys_per_sec":0,"num_objects_recovered":1258,"num_bytes_recovered":1258,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:36.333 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:36.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:36.673 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:36.694 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2024-09-16T02:50:35.714495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":17805,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14112,"num_bytes_recovered":79647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":319720,"kb_used_data":118536,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188423960,"statfs":{"total":193273528320,"available":192946135040,"internally_reserved":0,"allocated":121380864,"data_stored":351766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2516,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":1258,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003576"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":13454,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:34.013586+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:34.013586+0000","last_peered":"2024-09-16T02:50:34.013586+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:34.013586+0000","last_undegraded":"2024-09-16T02:50:34.013586+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1728,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3134,"num_bytes_recovered":3170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3133},{"shards":"2,3","objects":864}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":17805,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14112,"num_bytes_recovered":79647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":334667776,"data_stored":644995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171048,"kb_used_data":59352,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94200792,"statfs":{"total":96636764160,"available":96461611008,"internally_reserved":0,"allocated":60776448,"data_stored":175904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673224,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148672,"kb_used_data":59184,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94223168,"statfs":{"total":96636764160,"available":96484524032,"internally_reserved":0,"allocated":60604416,"data_stored":175862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986212,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60366848,"data_stored":80241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60923904,"data_stored":80377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:39.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:40.051 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:40.051 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:40.070 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":680,"stamp":"2024-09-16T02:50:39.833745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16341,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14844,"num_bytes_recovered":80379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":328860,"kb_used_data":123516,"kb_used_omap":3,"kb_used_meta":205308,"kb_avail":188414820,"statfs":{"total":193273528320,"available":192936775680,"internally_reserved":0,"allocated":126480384,"data_stored":353011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":210236298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2458,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1229,"num_bytes_recovered":1229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.121143"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":15650,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:39.021741+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:39.021741+0000","last_peered":"2024-09-16T02:50:39.021741+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:39.021741+0000","last_undegraded":"2024-09-16T02:50:39.021741+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":264,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3866,"num_bytes_recovered":3902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3865},{"shards":"2,3","objects":132}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16341,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14844,"num_bytes_recovered":80379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339881984,"data_stored":646268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173524,"kb_used_data":61828,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94198316,"statfs":{"total":96636764160,"available":96459075584,"internally_reserved":0,"allocated":63311872,"data_stored":176523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673225,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155336,"kb_used_data":61688,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94216504,"statfs":{"total":96636764160,"available":96477700096,"internally_reserved":0,"allocated":63168512,"data_stored":176488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986213,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62918656,"data_stored":80864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63586304,"data_stored":81027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:40.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:40.395 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:40.415 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":680,"stamp":"2024-09-16T02:50:39.833745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16341,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14844,"num_bytes_recovered":80379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":328860,"kb_used_data":123516,"kb_used_omap":3,"kb_used_meta":205308,"kb_avail":188414820,"statfs":{"total":193273528320,"available":192936775680,"internally_reserved":0,"allocated":126480384,"data_stored":353011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":210236298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2458,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1229,"num_bytes_recovered":1229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.121143"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":15650,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:39.021741+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:39.021741+0000","last_peered":"2024-09-16T02:50:39.021741+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:39.021741+0000","last_undegraded":"2024-09-16T02:50:39.021741+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":264,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3866,"num_bytes_recovered":3902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3865},{"shards":"2,3","objects":132}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16341,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14844,"num_bytes_recovered":80379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339881984,"data_stored":646268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173524,"kb_used_data":61828,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94198316,"statfs":{"total":96636764160,"available":96459075584,"internally_reserved":0,"allocated":63311872,"data_stored":176523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673225,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155336,"kb_used_data":61688,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94216504,"statfs":{"total":96636764160,"available":96477700096,"internally_reserved":0,"allocated":63168512,"data_stored":176488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986213,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62918656,"data_stored":80864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63586304,"data_stored":81027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:40.416 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:40.416 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:40.416 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:40.416 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:40.417 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:40.417 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:40.417 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-16T02:50:40.417 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "19'3998", 'reported_seq': 15650, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:39.021741+0000', 'last_change': '2024-09-16T02:49:45.380183+0000', 'last_active': '2024-09-16T02:50:39.021741+0000', 'last_peered': '2024-09-16T02:50:39.021741+0000', 'last_clean': '2024-09-16T02:46:37.435854+0000', 'last_became_active': '2024-09-16T02:47:26.514940+0000', 'last_became_peered': '2024-09-16T02:47:26.514940+0000', 'last_unstale': '2024-09-16T02:50:39.021741+0000', 'last_undegraded': '2024-09-16T02:50:39.021741+0000', 'last_fullsized': '2024-09-16T02:47:26.333672+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3998, 'log_dups_size': 0, 'ondisk_log_size': 3998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.004837016, 'stat_sum': {'num_bytes': 30215, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 7994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 264, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 4022, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3866, 'num_bytes_recovered': 3902, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3865}, {'shards': '2,3', 'objects': 132}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:40.417 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:40.417 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:40.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:40.841 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:40.841 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1300,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":336752640,"bytes_avail":192936775680,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":16341,"misplaced_total":63846,"misplaced_ratio":0.25594399022648245,"recovering_objects_per_sec":135,"recovering_bytes_per_sec":135,"recovering_keys_per_sec":0,"num_objects_recovered":1631,"num_bytes_recovered":1631,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:40.865 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:40.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:41.202 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:41.202 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:41.220 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2024-09-16T02:50:40.837907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16341,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14844,"num_bytes_recovered":80379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":328860,"kb_used_data":123516,"kb_used_omap":3,"kb_used_meta":205308,"kb_avail":188414820,"statfs":{"total":193273528320,"available":192936775680,"internally_reserved":0,"allocated":126480384,"data_stored":353011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":210236298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2458,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1229,"num_bytes_recovered":1229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.124693"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.805402+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:49:46.805402+0000","last_peered":"2024-09-16T02:49:46.805402+0000","last_clean":"2024-09-16T02:49:46.805402+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:49:46.805402+0000","last_undegraded":"2024-09-16T02:49:46.805402+0000","last_fullsized":"2024-09-16T02:49:46.805402+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4047,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.799678+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:49:46.799678+0000","last_peered":"2024-09-16T02:49:46.799678+0000","last_clean":"2024-09-16T02:49:46.799678+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:49:46.799678+0000","last_undegraded":"2024-09-16T02:49:46.799678+0000","last_fullsized":"2024-09-16T02:49:46.799678+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":15650,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:39.021741+0000","last_change":"2024-09-16T02:49:45.380183+0000","last_active":"2024-09-16T02:50:39.021741+0000","last_peered":"2024-09-16T02:50:39.021741+0000","last_clean":"2024-09-16T02:46:37.435854+0000","last_became_active":"2024-09-16T02:47:26.514940+0000","last_became_peered":"2024-09-16T02:47:26.514940+0000","last_unstale":"2024-09-16T02:50:39.021741+0000","last_undegraded":"2024-09-16T02:50:39.021741+0000","last_fullsized":"2024-09-16T02:47:26.333672+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":264,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3866,"num_bytes_recovered":3902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3865},{"shards":"2,3","objects":132}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3977,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-16T02:49:46.797182+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:49:46.797182+0000","last_peered":"2024-09-16T02:49:46.797182+0000","last_clean":"2024-09-16T02:49:46.797182+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:49:46.797182+0000","last_undegraded":"2024-09-16T02:49:46.797182+0000","last_fullsized":"2024-09-16T02:49:46.797182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16341,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14844,"num_bytes_recovered":80379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339881984,"data_stored":646268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173524,"kb_used_data":61828,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94198316,"statfs":{"total":96636764160,"available":96459075584,"internally_reserved":0,"allocated":63311872,"data_stored":176523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673225,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155336,"kb_used_data":61688,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94216504,"statfs":{"total":96636764160,"available":96477700096,"internally_reserved":0,"allocated":63168512,"data_stored":176488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986213,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62918656,"data_stored":80864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63586304,"data_stored":81027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:44.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:44.572 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:44.572 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:44.596 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2024-09-16T02:50:43.716818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16077,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14976,"num_bytes_recovered":106711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":325536,"kb_used_data":128512,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188418144,"statfs":{"total":193273528320,"available":192940179456,"internally_reserved":0,"allocated":131596288,"data_stored":417618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1728,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":864,"num_bytes_recovered":27064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002323"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16077,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14976,"num_bytes_recovered":106711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":344477696,"data_stored":699778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163500,"kb_used_data":64284,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94208340,"statfs":{"total":96636764160,"available":96469340160,"internally_reserved":0,"allocated":65826816,"data_stored":208816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162036,"kb_used_data":64228,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":94209804,"statfs":{"total":96636764160,"available":96470839296,"internally_reserved":0,"allocated":65769472,"data_stored":208802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986214,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65470464,"data_stored":107681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65630208,"data_stored":107720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:44.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:44.949 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:44.949 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:44.968 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2024-09-16T02:50:43.716818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16077,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14976,"num_bytes_recovered":106711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":325536,"kb_used_data":128512,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188418144,"statfs":{"total":193273528320,"available":192940179456,"internally_reserved":0,"allocated":131596288,"data_stored":417618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1728,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":864,"num_bytes_recovered":27064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002323"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16077,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14976,"num_bytes_recovered":106711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":344477696,"data_stored":699778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163500,"kb_used_data":64284,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94208340,"statfs":{"total":96636764160,"available":96469340160,"internally_reserved":0,"allocated":65826816,"data_stored":208816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162036,"kb_used_data":64228,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":94209804,"statfs":{"total":96636764160,"available":96470839296,"internally_reserved":0,"allocated":65769472,"data_stored":208802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986214,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65470464,"data_stored":107681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65630208,"data_stored":107720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:44.969 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:44.969 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4129, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807804+0000', 'last_change': '2024-09-16T02:47:26.522592+0000', 'last_active': '2024-09-16T02:49:46.807804+0000', 'last_peered': '2024-09-16T02:49:46.807804+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:49:46.807804+0000', 'last_undegraded': '2024-09-16T02:47:26.507801+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4060, 'num_objects_misplaced': 4060, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:44.969 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:44.969 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4029, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.810840+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:49:46.810840+0000', 'last_peered': '2024-09-16T02:49:46.810840+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:49:46.810840+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:44.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:44.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4086, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807818+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:49:46.807818+0000', 'last_peered': '2024-09-16T02:49:46.807818+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:49:46.807818+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:44.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:44.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:49:46.807721+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:49:46.807721+0000', 'last_peered': '2024-09-16T02:49:46.807721+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:49:46.807721+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:44.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:45.370 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16077/63846 objects degraded (25.181%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1304,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":4}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":333348864,"bytes_avail":192940179456,"bytes_total":193273528320,"degraded_objects":16077,"degraded_total":63846,"degraded_ratio":0.25180904050371206,"misplaced_objects":16077,"misplaced_total":63846,"misplaced_ratio":0.25180904050371206,"recovering_objects_per_sec":107,"recovering_bytes_per_sec":3382,"recovering_keys_per_sec":0,"num_objects_recovered":864,"num_bytes_recovered":27064,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:45.389 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:45.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:45.713 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:45.713 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:45.733 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2024-09-16T02:50:43.716818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16077,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14976,"num_bytes_recovered":106711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":325536,"kb_used_data":128512,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188418144,"statfs":{"total":193273528320,"available":192940179456,"internally_reserved":0,"allocated":131596288,"data_stored":417618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1728,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":864,"num_bytes_recovered":27064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002323"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4129,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807804+0000","last_change":"2024-09-16T02:47:26.522592+0000","last_active":"2024-09-16T02:49:46.807804+0000","last_peered":"2024-09-16T02:49:46.807804+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:49:46.807804+0000","last_undegraded":"2024-09-16T02:47:26.507801+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4060,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4060}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4029,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.810840+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:49:46.810840+0000","last_peered":"2024-09-16T02:49:46.810840+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:49:46.810840+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4086,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807818+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:49:46.807818+0000","last_peered":"2024-09-16T02:49:46.807818+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:49:46.807818+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:49:46.807721+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:49:46.807721+0000","last_peered":"2024-09-16T02:49:46.807721+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:49:46.807721+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16077,"num_objects_misplaced":16077,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14976,"num_bytes_recovered":106711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":344477696,"data_stored":699778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163500,"kb_used_data":64284,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94208340,"statfs":{"total":96636764160,"available":96469340160,"internally_reserved":0,"allocated":65826816,"data_stored":208816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162036,"kb_used_data":64228,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":94209804,"statfs":{"total":96636764160,"available":96470839296,"internally_reserved":0,"allocated":65769472,"data_stored":208802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986214,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65470464,"data_stored":107681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65630208,"data_stored":107720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82391040,"data_stored":203514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:48.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:49.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:49.100 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:49.120 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2024-09-16T02:50:47.717821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19723,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15183,"num_bytes_recovered":106918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":310788,"kb_used_data":130404,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188432892,"statfs":{"total":193273528320,"available":192955281408,"internally_reserved":0,"allocated":133533696,"data_stored":418091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4060,"num_objects_misplaced":3382,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":26539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002140"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4755,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:44.015529+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:44.015529+0000","last_peered":"2024-09-16T02:50:44.015529+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:44.015529+0000","last_undegraded":"2024-09-16T02:50:44.015529+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7706,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":207},{"shards":"2,3","objects":3853}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19723,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15183,"num_bytes_recovered":106918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329990144,"data_stored":670029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164432,"kb_used_data":65216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94207408,"statfs":{"total":96636764160,"available":96468385792,"internally_reserved":0,"allocated":66781184,"data_stored":209049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673227,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146356,"kb_used_data":65188,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94225484,"statfs":{"total":96636764160,"available":96486895616,"internally_reserved":0,"allocated":66752512,"data_stored":209042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986215,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66433024,"data_stored":107916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66576384,"data_stored":107951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:49.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:49.440 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:49.440 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:49.463 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2024-09-16T02:50:47.717821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19723,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15183,"num_bytes_recovered":106918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":310788,"kb_used_data":130404,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188432892,"statfs":{"total":193273528320,"available":192955281408,"internally_reserved":0,"allocated":133533696,"data_stored":418091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4060,"num_objects_misplaced":3382,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":26539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002140"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4755,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:44.015529+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:44.015529+0000","last_peered":"2024-09-16T02:50:44.015529+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:44.015529+0000","last_undegraded":"2024-09-16T02:50:44.015529+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7706,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":207},{"shards":"2,3","objects":3853}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19723,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15183,"num_bytes_recovered":106918,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329990144,"data_stored":670029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164432,"kb_used_data":65216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94207408,"statfs":{"total":96636764160,"available":96468385792,"internally_reserved":0,"allocated":66781184,"data_stored":209049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673227,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146356,"kb_used_data":65188,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94225484,"statfs":{"total":96636764160,"available":96486895616,"internally_reserved":0,"allocated":66752512,"data_stored":209042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986215,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66433024,"data_stored":107916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66576384,"data_stored":107951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:49.464 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:49.464 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 4755, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:44.015529+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:50:44.015529+0000', 'last_peered': '2024-09-16T02:50:44.015529+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:50:44.015529+0000', 'last_undegraded': '2024-09-16T02:50:44.015529+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7706, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 207, 'num_bytes_recovered': 207, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 207}, {'shards': '2,3', 'objects': 3853}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:49.464 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:49.464 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:49.464 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:49.464 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:49.465 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:49.465 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:49.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:49.884 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:49.885 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1309,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":318246912,"bytes_avail":192955281408,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":19235,"misplaced_total":63846,"misplaced_ratio":0.30127181029351879,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":2709,"recovering_keys_per_sec":0,"num_objects_recovered":583,"num_bytes_recovered":26783,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:49.907 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:49.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:50.219 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:50.219 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:50.238 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2024-09-16T02:50:49.718483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":310788,"kb_used_data":130404,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188432892,"statfs":{"total":193273528320,"available":192955281408,"internally_reserved":0,"allocated":133533696,"data_stored":418091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4060,"num_objects_misplaced":2894,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":26783,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.884737"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":5487,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:49.033999+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:49.033999+0000","last_peered":"2024-09-16T02:50:49.033999+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:49.033999+0000","last_undegraded":"2024-09-16T02:50:49.033999+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7218,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":451},{"shards":"2,3","objects":3609}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":329990144,"data_stored":670029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164432,"kb_used_data":65216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94207408,"statfs":{"total":96636764160,"available":96468385792,"internally_reserved":0,"allocated":66781184,"data_stored":209049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673227,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146356,"kb_used_data":65188,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94225484,"statfs":{"total":96636764160,"available":96486895616,"internally_reserved":0,"allocated":66752512,"data_stored":209042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986215,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":66433024,"data_stored":107916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66576384,"data_stored":107951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:53.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:53.609 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:53.610 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:53.630 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2024-09-16T02:50:51.719220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":311696,"kb_used_data":131312,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188431984,"statfs":{"total":193273528320,"available":192954351616,"internally_reserved":0,"allocated":134463488,"data_stored":418318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4060,"num_objects_misplaced":2894,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":26783,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.881313"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":5487,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:49.033999+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:49.033999+0000","last_peered":"2024-09-16T02:50:49.033999+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:49.033999+0000","last_undegraded":"2024-09-16T02:50:49.033999+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7218,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":451},{"shards":"2,3","objects":3609}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330915840,"data_stored":670255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164432,"kb_used_data":65216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94207408,"statfs":{"total":96636764160,"available":96468385792,"internally_reserved":0,"allocated":66781184,"data_stored":209049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673228,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147264,"kb_used_data":66096,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94224576,"statfs":{"total":96636764160,"available":96485965824,"internally_reserved":0,"allocated":67682304,"data_stored":209269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986216,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67358720,"data_stored":108142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66576384,"data_stored":107951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:53.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:53.957 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:53.958 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:53.980 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2024-09-16T02:50:53.719818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":316768,"kb_used_data":132224,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188426912,"statfs":{"total":193273528320,"available":192949157888,"internally_reserved":0,"allocated":135397376,"data_stored":418546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4060,"num_objects_misplaced":3158,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003656"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":5487,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:49.033999+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:49.033999+0000","last_peered":"2024-09-16T02:50:49.033999+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:49.033999+0000","last_undegraded":"2024-09-16T02:50:49.033999+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7218,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":451},{"shards":"2,3","objects":3609}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331845632,"data_stored":670482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":169504,"kb_used_data":66128,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94202336,"statfs":{"total":96636764160,"available":96463192064,"internally_reserved":0,"allocated":67715072,"data_stored":209277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673228,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147264,"kb_used_data":66096,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94224576,"statfs":{"total":96636764160,"available":96485965824,"internally_reserved":0,"allocated":67682304,"data_stored":209269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986216,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67358720,"data_stored":108142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67506176,"data_stored":108178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:53.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:53.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 5487, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:49.033999+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:50:49.033999+0000', 'last_peered': '2024-09-16T02:50:49.033999+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:50:49.033999+0000', 'last_undegraded': '2024-09-16T02:50:49.033999+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7218, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 451, 'num_bytes_recovered': 451, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 451}, {'shards': '2,3', 'objects': 3609}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:53.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:53.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:53.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:53.982 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:53.982 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:53.982 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:53.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:54.446 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:54.446 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1314,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":324370432,"bytes_avail":192949157888,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":19235,"misplaced_total":63846,"misplaced_ratio":0.30127181029351879,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":37,"recovering_keys_per_sec":0,"num_objects_recovered":451,"num_bytes_recovered":451,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:54.466 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:54.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:54.802 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:54.803 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:54.823 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2024-09-16T02:50:53.719818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":316768,"kb_used_data":132224,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188426912,"statfs":{"total":193273528320,"available":192949157888,"internally_reserved":0,"allocated":135397376,"data_stored":418546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4060,"num_objects_misplaced":3158,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003656"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":5487,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:49.033999+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:49.033999+0000","last_peered":"2024-09-16T02:50:49.033999+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:49.033999+0000","last_undegraded":"2024-09-16T02:50:49.033999+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7218,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":451},{"shards":"2,3","objects":3609}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":19235,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15427,"num_bytes_recovered":107162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331845632,"data_stored":670482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":169504,"kb_used_data":66128,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94202336,"statfs":{"total":96636764160,"available":96463192064,"internally_reserved":0,"allocated":67715072,"data_stored":209277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673228,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147264,"kb_used_data":66096,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94224576,"statfs":{"total":96636764160,"available":96485965824,"internally_reserved":0,"allocated":67682304,"data_stored":209269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986216,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67358720,"data_stored":108142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67506176,"data_stored":108178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:57.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:58.202 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:58.203 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:58.223 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2024-09-16T02:50:57.721113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15675,"num_bytes_recovered":107410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":318828,"kb_used_data":134284,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188424852,"statfs":{"total":193273528320,"available":192947048448,"internally_reserved":0,"allocated":137506816,"data_stored":419061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-984,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003727"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":6231,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:54.050777+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:54.050777+0000","last_peered":"2024-09-16T02:50:54.050777+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:54.050777+0000","last_undegraded":"2024-09-16T02:50:54.050777+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6722,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":699,"num_bytes_recovered":699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":699},{"shards":"2,3","objects":3361}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15675,"num_bytes_recovered":107410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":333950976,"data_stored":670996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170544,"kb_used_data":67168,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94201296,"statfs":{"total":96636764160,"available":96462127104,"internally_reserved":0,"allocated":68780032,"data_stored":209537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673229,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148284,"kb_used_data":67116,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94223556,"statfs":{"total":96636764160,"available":96484921344,"internally_reserved":0,"allocated":68726784,"data_stored":209524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986217,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68407296,"data_stored":108398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68562944,"data_stored":108436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:58.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:58.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:58.578 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:58.601 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2024-09-16T02:50:57.721113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15675,"num_bytes_recovered":107410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":318828,"kb_used_data":134284,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188424852,"statfs":{"total":193273528320,"available":192947048448,"internally_reserved":0,"allocated":137506816,"data_stored":419061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-984,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003727"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":6231,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:54.050777+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:54.050777+0000","last_peered":"2024-09-16T02:50:54.050777+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:54.050777+0000","last_undegraded":"2024-09-16T02:50:54.050777+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6722,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":699,"num_bytes_recovered":699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":699},{"shards":"2,3","objects":3361}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15675,"num_bytes_recovered":107410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":333950976,"data_stored":670996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170544,"kb_used_data":67168,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94201296,"statfs":{"total":96636764160,"available":96462127104,"internally_reserved":0,"allocated":68780032,"data_stored":209537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673229,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148284,"kb_used_data":67116,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94223556,"statfs":{"total":96636764160,"available":96484921344,"internally_reserved":0,"allocated":68726784,"data_stored":209524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986217,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68407296,"data_stored":108398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68562944,"data_stored":108436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:50:58.602 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:50:58.602 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 6231, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:54.050777+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:50:54.050777+0000', 'last_peered': '2024-09-16T02:50:54.050777+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:50:54.050777+0000', 'last_undegraded': '2024-09-16T02:50:54.050777+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6722, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 699, 'num_bytes_recovered': 699, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 699}, {'shards': '2,3', 'objects': 3361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:58.602 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:50:58.602 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:58.602 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:50:58.603 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:58.603 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:50:58.603 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:50:58.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:50:59.035 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:59.035 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1318,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":326479872,"bytes_avail":192947048448,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":18739,"misplaced_total":63846,"misplaced_ratio":0.29350311687498043,"recovering_objects_per_sec":40,"recovering_bytes_per_sec":40,"recovering_keys_per_sec":0,"num_objects_recovered":492,"num_bytes_recovered":492,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:50:59.056 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:50:59.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:50:59.380 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:50:59.380 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:50:59.399 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2024-09-16T02:50:57.721113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15675,"num_bytes_recovered":107410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":318828,"kb_used_data":134284,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188424852,"statfs":{"total":193273528320,"available":192947048448,"internally_reserved":0,"allocated":137506816,"data_stored":419061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-984,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":492,"num_bytes_recovered":492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003727"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":6231,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:54.050777+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:54.050777+0000","last_peered":"2024-09-16T02:50:54.050777+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:54.050777+0000","last_undegraded":"2024-09-16T02:50:54.050777+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6722,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":699,"num_bytes_recovered":699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":699},{"shards":"2,3","objects":3361}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18739,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15675,"num_bytes_recovered":107410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":333950976,"data_stored":670996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170544,"kb_used_data":67168,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94201296,"statfs":{"total":96636764160,"available":96462127104,"internally_reserved":0,"allocated":68780032,"data_stored":209537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673229,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":148284,"kb_used_data":67116,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94223556,"statfs":{"total":96636764160,"available":96484921344,"internally_reserved":0,"allocated":68726784,"data_stored":209524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986217,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68407296,"data_stored":108398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68562944,"data_stored":108436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:02.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:02.761 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:02.761 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:02.782 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2024-09-16T02:51:01.722522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18195,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15947,"num_bytes_recovered":107682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":323884,"kb_used_data":135180,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188419796,"statfs":{"total":193273528320,"available":192941871104,"internally_reserved":0,"allocated":138424320,"data_stored":419285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1040,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004039"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":7047,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:59.055318+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:59.055318+0000","last_peered":"2024-09-16T02:50:59.055318+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:59.055318+0000","last_undegraded":"2024-09-16T02:50:59.055318+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6178,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":971,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":971},{"shards":"2,3","objects":3089}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18195,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15947,"num_bytes_recovered":107682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335011840,"data_stored":671255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170544,"kb_used_data":67168,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94201296,"statfs":{"total":96636764160,"available":96462127104,"internally_reserved":0,"allocated":68780032,"data_stored":209537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673230,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153340,"kb_used_data":68012,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":69644288,"data_stored":209748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986218,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69468160,"data_stored":108657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68562944,"data_stored":108436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:02.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:03.112 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:03.112 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:03.133 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2024-09-16T02:51:01.722522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18195,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15947,"num_bytes_recovered":107682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":323884,"kb_used_data":135180,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188419796,"statfs":{"total":193273528320,"available":192941871104,"internally_reserved":0,"allocated":138424320,"data_stored":419285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1040,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004039"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":7047,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:59.055318+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:59.055318+0000","last_peered":"2024-09-16T02:50:59.055318+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:59.055318+0000","last_undegraded":"2024-09-16T02:50:59.055318+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6178,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":971,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":971},{"shards":"2,3","objects":3089}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18195,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15947,"num_bytes_recovered":107682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335011840,"data_stored":671255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170544,"kb_used_data":67168,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94201296,"statfs":{"total":96636764160,"available":96462127104,"internally_reserved":0,"allocated":68780032,"data_stored":209537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673230,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153340,"kb_used_data":68012,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":69644288,"data_stored":209748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986218,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69468160,"data_stored":108657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68562944,"data_stored":108436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:03.134 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:03.134 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 7047, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:50:59.055318+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:50:59.055318+0000', 'last_peered': '2024-09-16T02:50:59.055318+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:50:59.055318+0000', 'last_undegraded': '2024-09-16T02:50:59.055318+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6178, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 971, 'num_bytes_recovered': 971, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 971}, {'shards': '2,3', 'objects': 3089}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:03.134 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:03.135 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:03.135 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:03.135 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:03.135 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:03.135 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:03.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:03.549 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:03.550 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1323,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":331657216,"bytes_avail":192941871104,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":18195,"misplaced_total":63846,"misplaced_ratio":0.28498261441593836,"recovering_objects_per_sec":43,"recovering_bytes_per_sec":43,"recovering_keys_per_sec":0,"num_objects_recovered":520,"num_bytes_recovered":520,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:03.572 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:03.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:03.910 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:03.911 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:03.930 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":693,"stamp":"2024-09-16T02:51:03.723258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18195,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15947,"num_bytes_recovered":107682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":324936,"kb_used_data":136232,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188418744,"statfs":{"total":193273528320,"available":192940793856,"internally_reserved":0,"allocated":139501568,"data_stored":419548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1040,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004037"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":7047,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:50:59.055318+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:50:59.055318+0000","last_peered":"2024-09-16T02:50:59.055318+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:50:59.055318+0000","last_undegraded":"2024-09-16T02:50:59.055318+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6178,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":971,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":971},{"shards":"2,3","objects":3089}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":18195,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15947,"num_bytes_recovered":107682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":336089088,"data_stored":671518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640526,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171596,"kb_used_data":68220,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94200244,"statfs":{"total":96636764160,"available":96461049856,"internally_reserved":0,"allocated":69857280,"data_stored":209800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673230,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153340,"kb_used_data":68012,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":69644288,"data_stored":209748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986218,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69468160,"data_stored":108657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":108699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:06.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:07.302 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:07.302 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:07.322 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2024-09-16T02:51:05.723631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17615,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16237,"num_bytes_recovered":107972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":324936,"kb_used_data":136232,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188418744,"statfs":{"total":193273528320,"available":192940793856,"internally_reserved":0,"allocated":139501568,"data_stored":419548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1620,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":810,"num_bytes_recovered":810,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003813"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":7917,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:04.058232+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:04.058232+0000","last_peered":"2024-09-16T02:51:04.058232+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:04.058232+0000","last_undegraded":"2024-09-16T02:51:04.058232+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5598,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":1261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1261},{"shards":"2,3","objects":2799}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17615,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16237,"num_bytes_recovered":107972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":336089088,"data_stored":671518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640526,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171596,"kb_used_data":68220,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94200244,"statfs":{"total":96636764160,"available":96461049856,"internally_reserved":0,"allocated":69857280,"data_stored":209800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673230,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153340,"kb_used_data":68012,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":69644288,"data_stored":209748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986218,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69468160,"data_stored":108657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":108699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:07.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:07.638 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:07.638 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:07.658 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2024-09-16T02:51:05.723631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17615,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16237,"num_bytes_recovered":107972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":324936,"kb_used_data":136232,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188418744,"statfs":{"total":193273528320,"available":192940793856,"internally_reserved":0,"allocated":139501568,"data_stored":419548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1620,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":810,"num_bytes_recovered":810,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003813"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":7917,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:04.058232+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:04.058232+0000","last_peered":"2024-09-16T02:51:04.058232+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:04.058232+0000","last_undegraded":"2024-09-16T02:51:04.058232+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5598,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":1261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1261},{"shards":"2,3","objects":2799}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17615,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16237,"num_bytes_recovered":107972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":336089088,"data_stored":671518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640526,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171596,"kb_used_data":68220,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94200244,"statfs":{"total":96636764160,"available":96461049856,"internally_reserved":0,"allocated":69857280,"data_stored":209800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673230,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153340,"kb_used_data":68012,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":69644288,"data_stored":209748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986218,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69468160,"data_stored":108657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":108699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:07.659 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:07.659 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 7917, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:04.058232+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:51:04.058232+0000', 'last_peered': '2024-09-16T02:51:04.058232+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:51:04.058232+0000', 'last_undegraded': '2024-09-16T02:51:04.058232+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5598, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1261, 'num_bytes_recovered': 1261, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1261}, {'shards': '2,3', 'objects': 2799}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:07.659 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:07.659 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:07.659 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:07.659 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:07.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:07.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:07.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:08.070 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:08.071 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1327,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":334970880,"bytes_avail":192938557440,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":17615,"misplaced_total":63846,"misplaced_ratio":0.2758982551765185,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":46,"recovering_keys_per_sec":0,"num_objects_recovered":562,"num_bytes_recovered":562,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:08.091 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:08.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:08.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:08.420 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:08.439 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2024-09-16T02:51:07.724355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17615,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16237,"num_bytes_recovered":107972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":327120,"kb_used_data":138416,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188416560,"statfs":{"total":193273528320,"available":192938557440,"internally_reserved":0,"allocated":141737984,"data_stored":420094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1124,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":562,"num_bytes_recovered":562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004022"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":7917,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:04.058232+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:04.058232+0000","last_peered":"2024-09-16T02:51:04.058232+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:04.058232+0000","last_undegraded":"2024-09-16T02:51:04.058232+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5598,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":1261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1261},{"shards":"2,3","objects":2799}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17615,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16237,"num_bytes_recovered":107972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":338427904,"data_stored":672089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640527,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172688,"kb_used_data":69312,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94199152,"statfs":{"total":96636764160,"available":96459931648,"internally_reserved":0,"allocated":70975488,"data_stored":210073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673231,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":154432,"kb_used_data":69104,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94217408,"statfs":{"total":96636764160,"available":96478625792,"internally_reserved":0,"allocated":70762496,"data_stored":210021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986219,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70590464,"data_stored":108931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70856704,"data_stored":108996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:11.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:11.779 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:11.780 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:11.801 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":697,"stamp":"2024-09-16T02:51:11.725541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17045,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":108257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":328304,"kb_used_data":139600,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188415376,"statfs":{"total":193273528320,"available":192937345024,"internally_reserved":0,"allocated":142950400,"data_stored":420390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1150,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":575,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":8772,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:09.058588+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:09.058588+0000","last_peered":"2024-09-16T02:51:09.058588+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:09.058588+0000","last_undegraded":"2024-09-16T02:51:09.058588+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5028,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":1546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1546},{"shards":"2,3","objects":2514}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17045,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":108257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339648512,"data_stored":672387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640527,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172688,"kb_used_data":69312,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94199152,"statfs":{"total":96636764160,"available":96459931648,"internally_reserved":0,"allocated":70975488,"data_stored":210073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155616,"kb_used_data":70288,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94216224,"statfs":{"total":96636764160,"available":96477413376,"internally_reserved":0,"allocated":71974912,"data_stored":210317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986220,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71811072,"data_stored":109229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70856704,"data_stored":108996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:11.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:12.130 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:12.130 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:12.151 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":697,"stamp":"2024-09-16T02:51:11.725541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17045,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":108257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":328304,"kb_used_data":139600,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188415376,"statfs":{"total":193273528320,"available":192937345024,"internally_reserved":0,"allocated":142950400,"data_stored":420390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1150,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":575,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":8772,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:09.058588+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:09.058588+0000","last_peered":"2024-09-16T02:51:09.058588+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:09.058588+0000","last_undegraded":"2024-09-16T02:51:09.058588+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5028,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":1546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1546},{"shards":"2,3","objects":2514}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17045,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":108257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339648512,"data_stored":672387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640527,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172688,"kb_used_data":69312,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94199152,"statfs":{"total":96636764160,"available":96459931648,"internally_reserved":0,"allocated":70975488,"data_stored":210073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155616,"kb_used_data":70288,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94216224,"statfs":{"total":96636764160,"available":96477413376,"internally_reserved":0,"allocated":71974912,"data_stored":210317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986220,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71811072,"data_stored":109229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70856704,"data_stored":108996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:12.151 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:12.151 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 8772, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:09.058588+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:51:09.058588+0000', 'last_peered': '2024-09-16T02:51:09.058588+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:51:09.058588+0000', 'last_undegraded': '2024-09-16T02:51:09.058588+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5028, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1546, 'num_bytes_recovered': 1546, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1546}, {'shards': '2,3', 'objects': 2514}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:12.152 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:12.152 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:12.152 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:12.152 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:12.152 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:12.152 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:12.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:12.580 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1332,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":336183296,"bytes_avail":192937345024,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":17045,"misplaced_total":63846,"misplaced_ratio":0.2669705228205369,"recovering_objects_per_sec":47,"recovering_bytes_per_sec":47,"recovering_keys_per_sec":0,"num_objects_recovered":575,"num_bytes_recovered":575,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T02:48:37.679323+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:12.602 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:12.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:12.929 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:12.947 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":697,"stamp":"2024-09-16T02:51:11.725541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17045,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":108257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":328304,"kb_used_data":139600,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188415376,"statfs":{"total":193273528320,"available":192937345024,"internally_reserved":0,"allocated":142950400,"data_stored":420390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1150,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":575,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":8772,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:09.058588+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:09.058588+0000","last_peered":"2024-09-16T02:51:09.058588+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:09.058588+0000","last_undegraded":"2024-09-16T02:51:09.058588+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5028,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1546,"num_bytes_recovered":1546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1546},{"shards":"2,3","objects":2514}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":17045,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":108257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":339648512,"data_stored":672387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640527,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172688,"kb_used_data":69312,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94199152,"statfs":{"total":96636764160,"available":96459931648,"internally_reserved":0,"allocated":70975488,"data_stored":210073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155616,"kb_used_data":70288,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94216224,"statfs":{"total":96636764160,"available":96477413376,"internally_reserved":0,"allocated":71974912,"data_stored":210317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986220,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71811072,"data_stored":109229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70856704,"data_stored":108996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:15.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:16.286 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:16.287 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:16.307 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":699,"stamp":"2024-09-16T02:51:15.726660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":16431,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16829,"num_bytes_recovered":108564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":333736,"kb_used_data":140872,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188409944,"statfs":{"total":193273528320,"available":192931782656,"internally_reserved":0,"allocated":144252928,"data_stored":420708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1764,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":9693,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:14.058218+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:14.058218+0000","last_peered":"2024-09-16T02:51:14.058218+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:14.058218+0000","last_undegraded":"2024-09-16T02:51:14.058218+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4414,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":1853,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1853},{"shards":"2,3","objects":2207}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":16431,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16829,"num_bytes_recovered":108564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":340992000,"data_stored":672715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178120,"kb_used_data":70584,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94193720,"statfs":{"total":96636764160,"available":96454369280,"internally_reserved":0,"allocated":72278016,"data_stored":210391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155616,"kb_used_data":70288,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94216224,"statfs":{"total":96636764160,"available":96477413376,"internally_reserved":0,"allocated":71974912,"data_stored":210317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986220,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71811072,"data_stored":109229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":72200192,"data_stored":109324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:16.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:16.634 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:16.634 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:16.656 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":699,"stamp":"2024-09-16T02:51:15.726660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":16431,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16829,"num_bytes_recovered":108564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":333736,"kb_used_data":140872,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188409944,"statfs":{"total":193273528320,"available":192931782656,"internally_reserved":0,"allocated":144252928,"data_stored":420708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1764,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":9693,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:14.058218+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:14.058218+0000","last_peered":"2024-09-16T02:51:14.058218+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:14.058218+0000","last_undegraded":"2024-09-16T02:51:14.058218+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4414,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":1853,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1853},{"shards":"2,3","objects":2207}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":16431,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16829,"num_bytes_recovered":108564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":340992000,"data_stored":672715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178120,"kb_used_data":70584,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94193720,"statfs":{"total":96636764160,"available":96454369280,"internally_reserved":0,"allocated":72278016,"data_stored":210391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155616,"kb_used_data":70288,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94216224,"statfs":{"total":96636764160,"available":96477413376,"internally_reserved":0,"allocated":71974912,"data_stored":210317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986220,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71811072,"data_stored":109229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":72200192,"data_stored":109324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:16.657 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:16.657 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 9693, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:14.058218+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:51:14.058218+0000', 'last_peered': '2024-09-16T02:51:14.058218+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:51:14.058218+0000', 'last_undegraded': '2024-09-16T02:51:14.058218+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4414, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1853, 'num_bytes_recovered': 1853, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1853}, {'shards': '2,3', 'objects': 2207}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:16.657 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:16.657 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:16.658 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:16.658 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:16.658 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:16.658 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:16.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:17.068 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:17.068 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1336,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":341745664,"bytes_avail":192931782656,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":16431,"misplaced_total":63846,"misplaced_ratio":0.25735363217742696,"recovering_objects_per_sec":73,"recovering_bytes_per_sec":73,"recovering_keys_per_sec":0,"num_objects_recovered":882,"num_bytes_recovered":882,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-16T02:51:13.726087+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:51:17.089 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:17.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:17.407 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:17.408 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:17.426 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":699,"stamp":"2024-09-16T02:51:15.726660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":16431,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16829,"num_bytes_recovered":108564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":333736,"kb_used_data":140872,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188409944,"statfs":{"total":193273528320,"available":192931782656,"internally_reserved":0,"allocated":144252928,"data_stored":420708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1764,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":9693,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:14.058218+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:14.058218+0000","last_peered":"2024-09-16T02:51:14.058218+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:14.058218+0000","last_undegraded":"2024-09-16T02:51:14.058218+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4414,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":1853,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1853},{"shards":"2,3","objects":2207}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":16431,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16829,"num_bytes_recovered":108564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":340992000,"data_stored":672715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178120,"kb_used_data":70584,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94193720,"statfs":{"total":96636764160,"available":96454369280,"internally_reserved":0,"allocated":72278016,"data_stored":210391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673232,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155616,"kb_used_data":70288,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94216224,"statfs":{"total":96636764160,"available":96477413376,"internally_reserved":0,"allocated":71974912,"data_stored":210317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986220,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71811072,"data_stored":109229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":72200192,"data_stored":109324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:20.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:20.784 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:20.784 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:20.805 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2024-09-16T02:51:19.728273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":336424,"kb_used_data":143560,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188407256,"statfs":{"total":193273528320,"available":192929030144,"internally_reserved":0,"allocated":147005440,"data_stored":421380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1898,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":949,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003917"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":10764,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:19.065782+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:19.065782+0000","last_peered":"2024-09-16T02:51:19.065782+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:19.065782+0000","last_undegraded":"2024-09-16T02:51:19.065782+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3700,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":2210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2210},{"shards":"2,3","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":343891968,"data_stored":673423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179556,"kb_used_data":72020,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94192284,"statfs":{"total":96636764160,"available":96452898816,"internally_reserved":0,"allocated":73748480,"data_stored":210750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":156868,"kb_used_data":71540,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94214972,"statfs":{"total":96636764160,"available":96476131328,"internally_reserved":0,"allocated":73256960,"data_stored":210630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986221,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73224192,"data_stored":109574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73687040,"data_stored":109687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:20.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:21.139 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:21.139 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:21.159 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2024-09-16T02:51:19.728273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":336424,"kb_used_data":143560,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188407256,"statfs":{"total":193273528320,"available":192929030144,"internally_reserved":0,"allocated":147005440,"data_stored":421380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1898,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":949,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003917"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":10764,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:19.065782+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:19.065782+0000","last_peered":"2024-09-16T02:51:19.065782+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:19.065782+0000","last_undegraded":"2024-09-16T02:51:19.065782+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3700,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":2210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2210},{"shards":"2,3","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3981,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841876+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841876+0000","last_peered":"2024-09-16T02:50:40.841876+0000","last_clean":"2024-09-16T02:50:40.841876+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841876+0000","last_undegraded":"2024-09-16T02:50:40.841876+0000","last_fullsized":"2024-09-16T02:50:40.841876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":343891968,"data_stored":673423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179556,"kb_used_data":72020,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94192284,"statfs":{"total":96636764160,"available":96452898816,"internally_reserved":0,"allocated":73748480,"data_stored":210750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":156868,"kb_used_data":71540,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94214972,"statfs":{"total":96636764160,"available":96476131328,"internally_reserved":0,"allocated":73256960,"data_stored":210630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986221,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73224192,"data_stored":109574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73687040,"data_stored":109687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:21.160 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:21.160 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 10764, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:19.065782+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:51:19.065782+0000', 'last_peered': '2024-09-16T02:51:19.065782+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:51:19.065782+0000', 'last_undegraded': '2024-09-16T02:51:19.065782+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3700, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2210, 'num_bytes_recovered': 2210, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2210}, {'shards': '2,3', 'objects': 1850}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:21.160 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:21.160 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:21.160 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:21.161 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:21.161 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:21.161 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:21.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:21.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:21.580 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1341,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":344498176,"bytes_avail":192929030144,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":15717,"misplaced_total":63846,"misplaced_ratio":0.2461704726999342,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":79,"recovering_keys_per_sec":0,"num_objects_recovered":949,"num_bytes_recovered":949,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-16T02:51:13.726087+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T02:51:21.603 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:21.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:21.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:21.938 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:21.959 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2024-09-16T02:51:21.729076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":342136,"kb_used_data":145112,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188401544,"statfs":{"total":193273528320,"available":192923181056,"internally_reserved":0,"allocated":148594688,"data_stored":421768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1328,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004127"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":10764,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:19.065782+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:19.065782+0000","last_peered":"2024-09-16T02:51:19.065782+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:19.065782+0000","last_undegraded":"2024-09-16T02:51:19.065782+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3700,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":2210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2210},{"shards":"2,3","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":345546752,"data_stored":673827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179556,"kb_used_data":72020,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94192284,"statfs":{"total":96636764160,"available":96452898816,"internally_reserved":0,"allocated":73748480,"data_stored":210750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162580,"kb_used_data":73092,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94209260,"statfs":{"total":96636764160,"available":96470282240,"internally_reserved":0,"allocated":74846208,"data_stored":211018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986222,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74878976,"data_stored":109978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73687040,"data_stored":109687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:24.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:25.312 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:25.313 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:25.333 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-09-16T02:51:23.729560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":343744,"kb_used_data":146720,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188399936,"statfs":{"total":193273528320,"available":192921534464,"internally_reserved":0,"allocated":150241280,"data_stored":422170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1328,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004018"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":10764,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:19.065782+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:19.065782+0000","last_peered":"2024-09-16T02:51:19.065782+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:19.065782+0000","last_undegraded":"2024-09-16T02:51:19.065782+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3700,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":2210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2210},{"shards":"2,3","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347136000,"data_stored":674215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640530,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181164,"kb_used_data":73628,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94190676,"statfs":{"total":96636764160,"available":96451252224,"internally_reserved":0,"allocated":75395072,"data_stored":211152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162580,"kb_used_data":73092,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94209260,"statfs":{"total":96636764160,"available":96470282240,"internally_reserved":0,"allocated":74846208,"data_stored":211018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986222,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74878976,"data_stored":109978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75276288,"data_stored":110075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:25.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:25.663 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:25.663 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:25.683 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-09-16T02:51:23.729560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":343744,"kb_used_data":146720,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188399936,"statfs":{"total":193273528320,"available":192921534464,"internally_reserved":0,"allocated":150241280,"data_stored":422170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1328,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004018"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":10764,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:19.065782+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:19.065782+0000","last_peered":"2024-09-16T02:51:19.065782+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:19.065782+0000","last_undegraded":"2024-09-16T02:51:19.065782+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3700,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":2210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2210},{"shards":"2,3","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4094,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.841170+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.841170+0000","last_peered":"2024-09-16T02:50:40.841170+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.841170+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":15717,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17186,"num_bytes_recovered":108921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347136000,"data_stored":674215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640530,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181164,"kb_used_data":73628,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94190676,"statfs":{"total":96636764160,"available":96451252224,"internally_reserved":0,"allocated":75395072,"data_stored":211152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162580,"kb_used_data":73092,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94209260,"statfs":{"total":96636764160,"available":96470282240,"internally_reserved":0,"allocated":74846208,"data_stored":211018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986222,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74878976,"data_stored":109978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75276288,"data_stored":110075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:25.684 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:25.684 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 10764, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:19.065782+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:51:19.065782+0000', 'last_peered': '2024-09-16T02:51:19.065782+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:51:19.065782+0000', 'last_undegraded': '2024-09-16T02:51:19.065782+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3700, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2210, 'num_bytes_recovered': 2210, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2210}, {'shards': '2,3', 'objects': 1850}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:25.684 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:25.684 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:25.685 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:25.685 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:25.685 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:25.685 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4094, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.841170+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.841170+0000', 'last_peered': '2024-09-16T02:50:40.841170+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.841170+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:25.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:26.122 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1345,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":351993856,"bytes_avail":192921534464,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":14919,"misplaced_total":63846,"misplaced_ratio":0.23367164740156002,"recovering_objects_per_sec":88,"recovering_bytes_per_sec":88,"recovering_keys_per_sec":0,"num_objects_recovered":1063,"num_bytes_recovered":1063,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:26.144 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:26.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:26.485 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:26.485 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:26.506 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":704,"stamp":"2024-09-16T02:51:25.730322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":14919,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17585,"num_bytes_recovered":109320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":343744,"kb_used_data":146720,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188399936,"statfs":{"total":193273528320,"available":192921534464,"internally_reserved":0,"allocated":150241280,"data_stored":422170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":1063,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004306"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":11961,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:24.078828+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:24.078828+0000","last_peered":"2024-09-16T02:51:24.078828+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:24.078828+0000","last_undegraded":"2024-09-16T02:51:24.078828+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2902,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2609,"num_bytes_recovered":2609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2609},{"shards":"2,3","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":14919,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17585,"num_bytes_recovered":109320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347136000,"data_stored":674215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640530,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181164,"kb_used_data":73628,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94190676,"statfs":{"total":96636764160,"available":96451252224,"internally_reserved":0,"allocated":75395072,"data_stored":211152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162580,"kb_used_data":73092,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94209260,"statfs":{"total":96636764160,"available":96470282240,"internally_reserved":0,"allocated":74846208,"data_stored":211018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986222,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74878976,"data_stored":109978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75276288,"data_stored":110075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:29.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:29.860 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:29.861 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:29.879 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2024-09-16T02:51:29.731407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":351532,"kb_used_data":150348,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188392148,"statfs":{"total":193273528320,"available":192913559552,"internally_reserved":0,"allocated":153956352,"data_stored":423077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2450,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1225,"num_bytes_recovered":1225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003879"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":13368,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:29.084367+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:29.084367+0000","last_peered":"2024-09-16T02:51:29.084367+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:29.084367+0000","last_undegraded":"2024-09-16T02:51:29.084367+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3078,"num_bytes_recovered":3078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3078},{"shards":"2,3","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":350846976,"data_stored":675121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187148,"kb_used_data":75452,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94184692,"statfs":{"total":96636764160,"available":96445124608,"internally_reserved":0,"allocated":77262848,"data_stored":211608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673235,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164384,"kb_used_data":74896,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94207456,"statfs":{"total":96636764160,"available":96468434944,"internally_reserved":0,"allocated":76693504,"data_stored":211469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986223,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76759040,"data_stored":110437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77107200,"data_stored":110522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:29.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:30.216 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:30.217 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:30.238 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2024-09-16T02:51:29.731407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":351532,"kb_used_data":150348,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188392148,"statfs":{"total":193273528320,"available":192913559552,"internally_reserved":0,"allocated":153956352,"data_stored":423077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2450,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1225,"num_bytes_recovered":1225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003879"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":13368,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:29.084367+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:29.084367+0000","last_peered":"2024-09-16T02:51:29.084367+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:29.084367+0000","last_undegraded":"2024-09-16T02:51:29.084367+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3078,"num_bytes_recovered":3078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3078},{"shards":"2,3","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":350846976,"data_stored":675121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187148,"kb_used_data":75452,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94184692,"statfs":{"total":96636764160,"available":96445124608,"internally_reserved":0,"allocated":77262848,"data_stored":211608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673235,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164384,"kb_used_data":74896,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94207456,"statfs":{"total":96636764160,"available":96468434944,"internally_reserved":0,"allocated":76693504,"data_stored":211469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986223,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76759040,"data_stored":110437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77107200,"data_stored":110522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 13368, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:29.084367+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:51:29.084367+0000', 'last_peered': '2024-09-16T02:51:29.084367+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:51:29.084367+0000', 'last_undegraded': '2024-09-16T02:51:29.084367+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1964, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3078, 'num_bytes_recovered': 3078, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3078}, {'shards': '2,3', 'objects': 982}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:30.239 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4095, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.844973+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.844973+0000', 'last_peered': '2024-09-16T02:50:40.844973+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.844973+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:30.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:30.661 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:30.661 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1350,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":359968768,"bytes_avail":192913559552,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":13981,"misplaced_total":63846,"misplaced_ratio":0.21898004573504998,"recovering_objects_per_sec":102,"recovering_bytes_per_sec":102,"recovering_keys_per_sec":0,"num_objects_recovered":1225,"num_bytes_recovered":1225,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:30.682 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:30.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:31.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:31.011 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:31.031 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2024-09-16T02:51:29.731407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":351532,"kb_used_data":150348,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188392148,"statfs":{"total":193273528320,"available":192913559552,"internally_reserved":0,"allocated":153956352,"data_stored":423077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2450,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1225,"num_bytes_recovered":1225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003879"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4051,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842120+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842120+0000","last_peered":"2024-09-16T02:50:40.842120+0000","last_clean":"2024-09-16T02:50:40.842120+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842120+0000","last_undegraded":"2024-09-16T02:50:40.842120+0000","last_fullsized":"2024-09-16T02:50:40.842120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":13368,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:29.084367+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:29.084367+0000","last_peered":"2024-09-16T02:51:29.084367+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:29.084367+0000","last_undegraded":"2024-09-16T02:51:29.084367+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3078,"num_bytes_recovered":3078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3078},{"shards":"2,3","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":350846976,"data_stored":675121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187148,"kb_used_data":75452,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94184692,"statfs":{"total":96636764160,"available":96445124608,"internally_reserved":0,"allocated":77262848,"data_stored":211608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673235,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164384,"kb_used_data":74896,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94207456,"statfs":{"total":96636764160,"available":96468434944,"internally_reserved":0,"allocated":76693504,"data_stored":211469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986223,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":76759040,"data_stored":110437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77107200,"data_stored":110522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:34.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:34.390 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:34.390 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:34.411 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":708,"stamp":"2024-09-16T02:51:33.732559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":360148,"kb_used_data":154804,"kb_used_omap":3,"kb_used_meta":205308,"kb_avail":188383532,"statfs":{"total":193273528320,"available":192904736768,"internally_reserved":0,"allocated":158519296,"data_stored":424191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":210236298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1736,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003482"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4052,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842168+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842168+0000","last_peered":"2024-09-16T02:50:40.842168+0000","last_clean":"2024-09-16T02:50:40.842168+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842168+0000","last_undegraded":"2024-09-16T02:50:40.842168+0000","last_fullsized":"2024-09-16T02:50:40.842168+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":13368,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:29.084367+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:29.084367+0000","last_peered":"2024-09-16T02:51:29.084367+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:29.084367+0000","last_undegraded":"2024-09-16T02:51:29.084367+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3078,"num_bytes_recovered":3078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3078},{"shards":"2,3","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355487744,"data_stored":676254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189396,"kb_used_data":77700,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94182444,"statfs":{"total":96636764160,"available":96442822656,"internally_reserved":0,"allocated":79564800,"data_stored":212170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673236,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170752,"kb_used_data":77104,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94201088,"statfs":{"total":96636764160,"available":96461914112,"internally_reserved":0,"allocated":78954496,"data_stored":212021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986224,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79048704,"data_stored":110996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79458304,"data_stored":111096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:34.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:34.759 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:34.782 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":708,"stamp":"2024-09-16T02:51:33.732559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":360148,"kb_used_data":154804,"kb_used_omap":3,"kb_used_meta":205308,"kb_avail":188383532,"statfs":{"total":193273528320,"available":192904736768,"internally_reserved":0,"allocated":158519296,"data_stored":424191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":210236298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1736,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":868,"num_bytes_recovered":868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003482"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4052,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842168+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842168+0000","last_peered":"2024-09-16T02:50:40.842168+0000","last_clean":"2024-09-16T02:50:40.842168+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842168+0000","last_undegraded":"2024-09-16T02:50:40.842168+0000","last_fullsized":"2024-09-16T02:50:40.842168+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":13368,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:29.084367+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:29.084367+0000","last_peered":"2024-09-16T02:51:29.084367+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:29.084367+0000","last_undegraded":"2024-09-16T02:51:29.084367+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1964,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3078,"num_bytes_recovered":3078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3078},{"shards":"2,3","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4033,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839861+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.839861+0000","last_peered":"2024-09-16T02:50:40.839861+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.839861+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4090,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.839869+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.839869+0000","last_peered":"2024-09-16T02:50:40.839869+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.839869+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":13981,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18054,"num_bytes_recovered":109789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355487744,"data_stored":676254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189396,"kb_used_data":77700,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94182444,"statfs":{"total":96636764160,"available":96442822656,"internally_reserved":0,"allocated":79564800,"data_stored":212170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673236,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170752,"kb_used_data":77104,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94201088,"statfs":{"total":96636764160,"available":96461914112,"internally_reserved":0,"allocated":78954496,"data_stored":212021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986224,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79048704,"data_stored":110996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79458304,"data_stored":111096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "19'4060", 'reported_seq': 13368, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:29.084367+0000', 'last_change': '2024-09-16T02:50:39.798331+0000', 'last_active': '2024-09-16T02:51:29.084367+0000', 'last_peered': '2024-09-16T02:51:29.084367+0000', 'last_clean': '2024-09-16T02:46:39.818003+0000', 'last_became_active': '2024-09-16T02:47:26.515588+0000', 'last_became_peered': '2024-09-16T02:47:26.515588+0000', 'last_unstale': '2024-09-16T02:51:29.084367+0000', 'last_undegraded': '2024-09-16T02:51:29.084367+0000', 'last_fullsized': '2024-09-16T02:47:26.333784+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4060, 'log_dups_size': 0, 'ondisk_log_size': 4060, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000932713, 'stat_sum': {'num_bytes': 30283, 'num_objects': 4060, 'num_object_clones': 0, 'num_object_copies': 8120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1964, 'num_objects_unfound': 0, 'num_objects_dirty': 4060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4060, 'num_write_kb': 4084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3078, 'num_bytes_recovered': 3078, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3078}, {'shards': '2,3', 'objects': 982}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4033, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839861+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.839861+0000', 'last_peered': '2024-09-16T02:50:40.839861+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.839861+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4090, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.839869+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.839869+0000', 'last_peered': '2024-09-16T02:50:40.839869+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.839869+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:34.783 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4095, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.844973+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.844973+0000', 'last_peered': '2024-09-16T02:50:40.844973+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.844973+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:34.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:35.215 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:35.215 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1354,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":368791552,"bytes_avail":192904736768,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":13981,"misplaced_total":63846,"misplaced_ratio":0.21898004573504998,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":72,"recovering_keys_per_sec":0,"num_objects_recovered":868,"num_bytes_recovered":868,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:35.240 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:35.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:35.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:35.575 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:35.593 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-09-16T02:51:35.360112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":12713,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18688,"num_bytes_recovered":110423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":360148,"kb_used_data":154804,"kb_used_omap":3,"kb_used_meta":205308,"kb_avail":188383532,"statfs":{"total":193273528320,"available":192904736768,"internally_reserved":0,"allocated":158519296,"data_stored":424191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":210236298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3004,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":1502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.630551"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3994,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.840684+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:50:40.840684+0000","last_peered":"2024-09-16T02:50:40.840684+0000","last_clean":"2024-09-16T02:50:40.840684+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:50:40.840684+0000","last_undegraded":"2024-09-16T02:50:40.840684+0000","last_fullsized":"2024-09-16T02:50:40.840684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4052,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.842168+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:50:40.842168+0000","last_peered":"2024-09-16T02:50:40.842168+0000","last_clean":"2024-09-16T02:50:40.842168+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:50:40.842168+0000","last_undegraded":"2024-09-16T02:50:40.842168+0000","last_fullsized":"2024-09-16T02:50:40.842168+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":15271,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:34.092213+0000","last_change":"2024-09-16T02:50:39.798331+0000","last_active":"2024-09-16T02:51:34.092213+0000","last_peered":"2024-09-16T02:51:34.092213+0000","last_clean":"2024-09-16T02:46:39.818003+0000","last_became_active":"2024-09-16T02:47:26.515588+0000","last_became_peered":"2024-09-16T02:47:26.515588+0000","last_unstale":"2024-09-16T02:51:34.092213+0000","last_undegraded":"2024-09-16T02:51:34.092213+0000","last_fullsized":"2024-09-16T02:47:26.333784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":696,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3712,"num_bytes_recovered":3712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3712},{"shards":"2,3","objects":348}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4034,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.843049+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.843049+0000","last_peered":"2024-09-16T02:50:40.843049+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.843049+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4091,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.843539+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.843539+0000","last_peered":"2024-09-16T02:50:40.843539+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.843539+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4040,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.849994+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:50:40.849994+0000","last_peered":"2024-09-16T02:50:40.849994+0000","last_clean":"2024-09-16T02:50:40.849994+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:50:40.849994+0000","last_undegraded":"2024-09-16T02:50:40.849994+0000","last_fullsized":"2024-09-16T02:50:40.849994+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3982,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:50:40.841953+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:50:40.841953+0000","last_peered":"2024-09-16T02:50:40.841953+0000","last_clean":"2024-09-16T02:50:40.841953+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:50:40.841953+0000","last_undegraded":"2024-09-16T02:50:40.841953+0000","last_fullsized":"2024-09-16T02:50:40.841953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":12713,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18688,"num_bytes_recovered":110423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355487744,"data_stored":676254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189396,"kb_used_data":77700,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94182444,"statfs":{"total":96636764160,"available":96442822656,"internally_reserved":0,"allocated":79564800,"data_stored":212170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673236,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170752,"kb_used_data":77104,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94201088,"statfs":{"total":96636764160,"available":96461914112,"internally_reserved":0,"allocated":78954496,"data_stored":212021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986224,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79048704,"data_stored":110996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79458304,"data_stored":111096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:38.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:38.938 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:38.939 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:38.958 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":712,"stamp":"2024-09-16T02:51:37.733611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":12017,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19036,"num_bytes_recovered":136994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":353444,"kb_used_data":160580,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188390236,"statfs":{"total":193273528320,"available":192911601664,"internally_reserved":0,"allocated":164433920,"data_stored":486232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1964,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":982,"num_bytes_recovered":27205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002203"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4034,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.843049+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.843049+0000","last_peered":"2024-09-16T02:50:40.843049+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.843049+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4091,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.843539+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.843539+0000","last_peered":"2024-09-16T02:50:40.843539+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.843539+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":12017,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19036,"num_bytes_recovered":136994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":360898560,"data_stored":730009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179620,"kb_used_data":80404,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94192220,"statfs":{"total":96636764160,"available":96452833280,"internally_reserved":0,"allocated":82333696,"data_stored":244498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173824,"kb_used_data":80176,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94198016,"statfs":{"total":96636764160,"available":96458768384,"internally_reserved":0,"allocated":82100224,"data_stored":241734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986225,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81874944,"data_stored":137903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82042880,"data_stored":137944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:38.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:39.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:39.324 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:39.346 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":712,"stamp":"2024-09-16T02:51:37.733611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":12017,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19036,"num_bytes_recovered":136994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":353444,"kb_used_data":160580,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188390236,"statfs":{"total":193273528320,"available":192911601664,"internally_reserved":0,"allocated":164433920,"data_stored":486232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1964,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":982,"num_bytes_recovered":27205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002203"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4034,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.843049+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:50:40.843049+0000","last_peered":"2024-09-16T02:50:40.843049+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:50:40.843049+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4091,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.843539+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:50:40.843539+0000","last_peered":"2024-09-16T02:50:40.843539+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:50:40.843539+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4095,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:50:40.844973+0000","last_change":"2024-09-16T02:47:26.523183+0000","last_active":"2024-09-16T02:50:40.844973+0000","last_peered":"2024-09-16T02:50:40.844973+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:50:40.844973+0000","last_undegraded":"2024-09-16T02:47:26.499384+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4037,"num_objects_misplaced":4037,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4037}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12017,"num_objects_misplaced":12017,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19036,"num_bytes_recovered":136994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":360898560,"data_stored":730009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179620,"kb_used_data":80404,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94192220,"statfs":{"total":96636764160,"available":96452833280,"internally_reserved":0,"allocated":82333696,"data_stored":244498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173824,"kb_used_data":80176,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94198016,"statfs":{"total":96636764160,"available":96458768384,"internally_reserved":0,"allocated":82100224,"data_stored":241734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986225,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81874944,"data_stored":137903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82042880,"data_stored":137944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:39.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:39.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4034, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.843049+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:50:40.843049+0000', 'last_peered': '2024-09-16T02:50:40.843049+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:50:40.843049+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:39.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:39.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4091, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.843539+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:50:40.843539+0000', 'last_peered': '2024-09-16T02:50:40.843539+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:50:40.843539+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:39.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:39.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4095, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:50:40.844973+0000', 'last_change': '2024-09-16T02:47:26.523183+0000', 'last_active': '2024-09-16T02:50:40.844973+0000', 'last_peered': '2024-09-16T02:50:40.844973+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:50:40.844973+0000', 'last_undegraded': '2024-09-16T02:47:26.499384+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4037, 'num_objects_misplaced': 4037, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:39.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:39.733 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:39.733 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12017/63846 objects degraded (18.822%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1359,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":361926656,"bytes_avail":192911601664,"bytes_total":193273528320,"degraded_objects":12017,"degraded_total":63846,"degraded_ratio":0.18821852582777307,"misplaced_objects":12017,"misplaced_total":63846,"misplaced_ratio":0.18821852582777307,"recovering_objects_per_sec":122,"recovering_bytes_per_sec":3399,"recovering_keys_per_sec":0,"num_objects_recovered":982,"num_bytes_recovered":27205,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:39.754 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:39.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:40.079 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:40.079 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:40.095 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2024-09-16T02:51:39.734265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":353444,"kb_used_data":160580,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188390236,"statfs":{"total":193273528320,"available":192911601664,"internally_reserved":0,"allocated":164433920,"data_stored":486232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4037,"num_objects_misplaced":1747,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":27368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002419"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4589,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:39.084375+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:39.084375+0000","last_peered":"2024-09-16T02:51:39.084375+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:39.084375+0000","last_undegraded":"2024-09-16T02:51:39.084375+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7748,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":163},{"shards":"2,3","objects":3874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":360898560,"data_stored":730009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179620,"kb_used_data":80404,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94192220,"statfs":{"total":96636764160,"available":96452833280,"internally_reserved":0,"allocated":82333696,"data_stored":244498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173824,"kb_used_data":80176,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94198016,"statfs":{"total":96636764160,"available":96458768384,"internally_reserved":0,"allocated":82100224,"data_stored":241734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986225,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81874944,"data_stored":137903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82042880,"data_stored":137944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65994752,"data_stored":173299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:43.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:43.451 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:43.451 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:43.473 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":714,"stamp":"2024-09-16T02:51:41.734865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":341876,"kb_used_data":161492,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188401804,"statfs":{"total":193273528320,"available":192923447296,"internally_reserved":0,"allocated":165367808,"data_stored":489167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4037,"num_objects_misplaced":1747,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1145,"num_bytes_recovered":27368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002306"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4589,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:39.084375+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:39.084375+0000","last_peered":"2024-09-16T02:51:39.084375+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:39.084375+0000","last_undegraded":"2024-09-16T02:51:39.084375+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7748,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":163},{"shards":"2,3","objects":3874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":345153536,"data_stored":699948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179620,"kb_used_data":80404,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94192220,"statfs":{"total":96636764160,"available":96452833280,"internally_reserved":0,"allocated":82333696,"data_stored":244498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673238,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162256,"kb_used_data":81088,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94209584,"statfs":{"total":96636764160,"available":96470614016,"internally_reserved":0,"allocated":83034112,"data_stored":244669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986226,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82784256,"data_stored":138125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82042880,"data_stored":137944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:43.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:43.798 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:43.798 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:43.819 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-09-16T02:51:43.735681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":342736,"kb_used_data":162352,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188400944,"statfs":{"total":193273528320,"available":192922566656,"internally_reserved":0,"allocated":166248448,"data_stored":489382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4037,"num_objects_misplaced":3015,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":26734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.375569"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4589,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:39.084375+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:39.084375+0000","last_peered":"2024-09-16T02:51:39.084375+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:39.084375+0000","last_undegraded":"2024-09-16T02:51:39.084375+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7748,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":163},{"shards":"2,3","objects":3874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346038272,"data_stored":700164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640534,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180480,"kb_used_data":81264,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94191360,"statfs":{"total":96636764160,"available":96451952640,"internally_reserved":0,"allocated":83214336,"data_stored":244713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673238,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162256,"kb_used_data":81088,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94209584,"statfs":{"total":96636764160,"available":96470614016,"internally_reserved":0,"allocated":83034112,"data_stored":244669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986226,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82784256,"data_stored":138125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82927616,"data_stored":138160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:43.820 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:43.820 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:43.820 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:43.820 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:43.820 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:43.820 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 4589, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:39.084375+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:51:39.084375+0000', 'last_peered': '2024-09-16T02:51:39.084375+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:51:39.084375+0000', 'last_undegraded': '2024-09-16T02:51:39.084375+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7748, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 163, 'num_bytes_recovered': 163, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 163}, {'shards': '2,3', 'objects': 3874}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:43.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:44.227 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1363,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":350961664,"bytes_avail":192922566656,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":15728,"misplaced_total":63846,"misplaced_ratio":0.24634276227171634,"recovering_objects_per_sec":61,"recovering_bytes_per_sec":3191,"recovering_keys_per_sec":0,"num_objects_recovered":511,"num_bytes_recovered":26734,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:44.249 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:44.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:44.563 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:44.563 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:44.583 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-09-16T02:51:43.735681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":342736,"kb_used_data":162352,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188400944,"statfs":{"total":193273528320,"available":192922566656,"internally_reserved":0,"allocated":166248448,"data_stored":489382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4037,"num_objects_misplaced":3015,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":26734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.375569"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4589,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:39.084375+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:39.084375+0000","last_peered":"2024-09-16T02:51:39.084375+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:39.084375+0000","last_undegraded":"2024-09-16T02:51:39.084375+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7748,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":163},{"shards":"2,3","objects":3874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15728,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19199,"num_bytes_recovered":137157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":346038272,"data_stored":700164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640534,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180480,"kb_used_data":81264,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94191360,"statfs":{"total":96636764160,"available":96451952640,"internally_reserved":0,"allocated":83214336,"data_stored":244713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673238,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162256,"kb_used_data":81088,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94209584,"statfs":{"total":96636764160,"available":96470614016,"internally_reserved":0,"allocated":83034112,"data_stored":244669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986226,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":82784256,"data_stored":138125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82927616,"data_stored":138160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:47.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:47.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:47.938 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:47.957 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-09-16T02:51:47.736803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19439,"num_bytes_recovered":137397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":344588,"kb_used_data":164204,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188399092,"statfs":{"total":193273528320,"available":192920670208,"internally_reserved":0,"allocated":168144896,"data_stored":489845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4037,"num_objects_misplaced":2535,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":26974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.367183"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":5309,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:44.087793+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:44.087793+0000","last_peered":"2024-09-16T02:51:44.087793+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:44.087793+0000","last_undegraded":"2024-09-16T02:51:44.087793+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7268,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":403},{"shards":"2,3","objects":3634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19439,"num_bytes_recovered":137397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347926528,"data_stored":700625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640535,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181416,"kb_used_data":82200,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94190424,"statfs":{"total":96636764160,"available":96450994176,"internally_reserved":0,"allocated":84172800,"data_stored":244947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673239,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163172,"kb_used_data":82004,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94208668,"statfs":{"total":96636764160,"available":96469676032,"internally_reserved":0,"allocated":83972096,"data_stored":244898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83726336,"data_stored":138355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83873792,"data_stored":138391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:47.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:48.336 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:48.336 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:48.361 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-09-16T02:51:47.736803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19439,"num_bytes_recovered":137397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":344588,"kb_used_data":164204,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188399092,"statfs":{"total":193273528320,"available":192920670208,"internally_reserved":0,"allocated":168144896,"data_stored":489845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4037,"num_objects_misplaced":2535,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":26974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.367183"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":5309,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:44.087793+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:44.087793+0000","last_peered":"2024-09-16T02:51:44.087793+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:44.087793+0000","last_undegraded":"2024-09-16T02:51:44.087793+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7268,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":403},{"shards":"2,3","objects":3634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19439,"num_bytes_recovered":137397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347926528,"data_stored":700625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640535,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181416,"kb_used_data":82200,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94190424,"statfs":{"total":96636764160,"available":96450994176,"internally_reserved":0,"allocated":84172800,"data_stored":244947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673239,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163172,"kb_used_data":82004,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94208668,"statfs":{"total":96636764160,"available":96469676032,"internally_reserved":0,"allocated":83972096,"data_stored":244898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83726336,"data_stored":138355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83873792,"data_stored":138391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:48.363 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:48.363 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:48.363 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:48.363 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:48.364 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:48.364 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 5309, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:44.087793+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:51:44.087793+0000', 'last_peered': '2024-09-16T02:51:44.087793+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:51:44.087793+0000', 'last_undegraded': '2024-09-16T02:51:44.087793+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7268, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 403, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 403}, {'shards': '2,3', 'objects': 3634}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:48.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:48.782 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:48.782 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1368,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":352858112,"bytes_avail":192920670208,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":15248,"misplaced_total":63846,"misplaced_ratio":0.23882467186667922,"recovering_objects_per_sec":66,"recovering_bytes_per_sec":2372,"recovering_keys_per_sec":0,"num_objects_recovered":751,"num_bytes_recovered":26974,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:48.805 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:48.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:49.166 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:49.167 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:49.186 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-09-16T02:51:47.736803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19439,"num_bytes_recovered":137397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":344588,"kb_used_data":164204,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188399092,"statfs":{"total":193273528320,"available":192920670208,"internally_reserved":0,"allocated":168144896,"data_stored":489845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4037,"num_objects_misplaced":2535,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":26974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.367183"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":5309,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:44.087793+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:44.087793+0000","last_peered":"2024-09-16T02:51:44.087793+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:44.087793+0000","last_undegraded":"2024-09-16T02:51:44.087793+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7268,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":403},{"shards":"2,3","objects":3634}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":15248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19439,"num_bytes_recovered":137397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347926528,"data_stored":700625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640535,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181416,"kb_used_data":82200,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94190424,"statfs":{"total":96636764160,"available":96450994176,"internally_reserved":0,"allocated":84172800,"data_stored":244947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673239,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163172,"kb_used_data":82004,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94208668,"statfs":{"total":96636764160,"available":96469676032,"internally_reserved":0,"allocated":83972096,"data_stored":244898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83726336,"data_stored":138355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83873792,"data_stored":138391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:52.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:52.546 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:52.546 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:52.567 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2024-09-16T02:51:51.737997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14758,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19684,"num_bytes_recovered":137642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":345524,"kb_used_data":165136,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188398156,"statfs":{"total":193273528320,"available":192919711744,"internally_reserved":0,"allocated":169099264,"data_stored":490078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-970,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":485,"num_bytes_recovered":485,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003731"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":6044,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:49.095618+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:49.095618+0000","last_peered":"2024-09-16T02:51:49.095618+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:49.095618+0000","last_undegraded":"2024-09-16T02:51:49.095618+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6778,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":648},{"shards":"2,3","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14758,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19684,"num_bytes_recovered":137642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348905472,"data_stored":700864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640535,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181416,"kb_used_data":82200,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94190424,"statfs":{"total":96636764160,"available":96450994176,"internally_reserved":0,"allocated":84172800,"data_stored":244947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673240,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164108,"kb_used_data":82936,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94207732,"statfs":{"total":96636764160,"available":96468717568,"internally_reserved":0,"allocated":84926464,"data_stored":245131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986228,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84705280,"data_stored":138594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83873792,"data_stored":138391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:52.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:52.898 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:52.898 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:52.918 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2024-09-16T02:51:51.737997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14758,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19684,"num_bytes_recovered":137642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":345524,"kb_used_data":165136,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188398156,"statfs":{"total":193273528320,"available":192919711744,"internally_reserved":0,"allocated":169099264,"data_stored":490078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-970,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":485,"num_bytes_recovered":485,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003731"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":6044,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:49.095618+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:49.095618+0000","last_peered":"2024-09-16T02:51:49.095618+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:49.095618+0000","last_undegraded":"2024-09-16T02:51:49.095618+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6778,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":648},{"shards":"2,3","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14758,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19684,"num_bytes_recovered":137642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348905472,"data_stored":700864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640535,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181416,"kb_used_data":82200,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94190424,"statfs":{"total":96636764160,"available":96450994176,"internally_reserved":0,"allocated":84172800,"data_stored":244947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673240,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164108,"kb_used_data":82936,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94207732,"statfs":{"total":96636764160,"available":96468717568,"internally_reserved":0,"allocated":84926464,"data_stored":245131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986228,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84705280,"data_stored":138594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83873792,"data_stored":138391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:52.919 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:52.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:52.919 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:52.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:52.919 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:52.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 6044, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:49.095618+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:51:49.095618+0000', 'last_peered': '2024-09-16T02:51:49.095618+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:51:49.095618+0000', 'last_undegraded': '2024-09-16T02:51:49.095618+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6778, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 648, 'num_bytes_recovered': 648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 648}, {'shards': '2,3', 'objects': 3389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:52.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:53.302 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:53.302 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1372,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":353816576,"bytes_avail":192919711744,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":14758,"misplaced_total":63846,"misplaced_ratio":0.23114995457820381,"recovering_objects_per_sec":40,"recovering_bytes_per_sec":40,"recovering_keys_per_sec":0,"num_objects_recovered":485,"num_bytes_recovered":485,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:53.325 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:53.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:53.691 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:53.692 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:53.714 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2024-09-16T02:51:51.737997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14758,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19684,"num_bytes_recovered":137642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":345524,"kb_used_data":165136,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188398156,"statfs":{"total":193273528320,"available":192919711744,"internally_reserved":0,"allocated":169099264,"data_stored":490078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-970,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":485,"num_bytes_recovered":485,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003731"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":6044,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:49.095618+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:49.095618+0000","last_peered":"2024-09-16T02:51:49.095618+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:49.095618+0000","last_undegraded":"2024-09-16T02:51:49.095618+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6778,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":648},{"shards":"2,3","objects":3389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14758,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19684,"num_bytes_recovered":137642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348905472,"data_stored":700864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640535,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181416,"kb_used_data":82200,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94190424,"statfs":{"total":96636764160,"available":96450994176,"internally_reserved":0,"allocated":84172800,"data_stored":244947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673240,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164108,"kb_used_data":82936,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94207732,"statfs":{"total":96636764160,"available":96468717568,"internally_reserved":0,"allocated":84926464,"data_stored":245131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986228,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84705280,"data_stored":138594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83873792,"data_stored":138391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:56.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:57.060 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:57.060 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:57.080 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2024-09-16T02:51:55.739135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19939,"num_bytes_recovered":137897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":350652,"kb_used_data":166104,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188393028,"statfs":{"total":193273528320,"available":192914460672,"internally_reserved":0,"allocated":170090496,"data_stored":490320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1480,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":6809,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:54.093800+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:54.093800+0000","last_peered":"2024-09-16T02:51:54.093800+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:54.093800+0000","last_undegraded":"2024-09-16T02:51:54.093800+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6268,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":903,"num_bytes_recovered":903,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":903},{"shards":"2,3","objects":3134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19939,"num_bytes_recovered":137897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":349929472,"data_stored":701114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640536,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186544,"kb_used_data":83168,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94185296,"statfs":{"total":96636764160,"available":96445743104,"internally_reserved":0,"allocated":85164032,"data_stored":245189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673240,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164108,"kb_used_data":82936,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94207732,"statfs":{"total":96636764160,"available":96468717568,"internally_reserved":0,"allocated":84926464,"data_stored":245131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986228,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84705280,"data_stored":138594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":84897792,"data_stored":138641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:57.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:57.418 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:57.418 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:57.438 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2024-09-16T02:51:55.739135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19939,"num_bytes_recovered":137897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":350652,"kb_used_data":166104,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188393028,"statfs":{"total":193273528320,"available":192914460672,"internally_reserved":0,"allocated":170090496,"data_stored":490320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1480,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3998,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374700+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374700+0000","last_peered":"2024-09-16T02:51:36.374700+0000","last_clean":"2024-09-16T02:51:36.374700+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374700+0000","last_undegraded":"2024-09-16T02:51:36.374700+0000","last_fullsized":"2024-09-16T02:51:36.374700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":6809,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:54.093800+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:54.093800+0000","last_peered":"2024-09-16T02:51:54.093800+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:54.093800+0000","last_undegraded":"2024-09-16T02:51:54.093800+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6268,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":903,"num_bytes_recovered":903,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":903},{"shards":"2,3","objects":3134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19939,"num_bytes_recovered":137897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":349929472,"data_stored":701114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640536,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":186544,"kb_used_data":83168,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94185296,"statfs":{"total":96636764160,"available":96445743104,"internally_reserved":0,"allocated":85164032,"data_stored":245189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673240,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164108,"kb_used_data":82936,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94207732,"statfs":{"total":96636764160,"available":96468717568,"internally_reserved":0,"allocated":84926464,"data_stored":245131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986228,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84705280,"data_stored":138594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":84897792,"data_stored":138641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:51:57.439 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:51:57.439 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:57.439 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:51:57.439 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:57.439 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:51:57.439 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 6809, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:54.093800+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:51:54.093800+0000', 'last_peered': '2024-09-16T02:51:54.093800+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:51:54.093800+0000', 'last_undegraded': '2024-09-16T02:51:54.093800+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6268, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 903, 'num_bytes_recovered': 903, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 903}, {'shards': '2,3', 'objects': 3134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:51:57.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:51:57.845 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:57.845 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1377,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":361230336,"bytes_avail":192912297984,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":14248,"misplaced_total":63846,"misplaced_ratio":0.22316198352285185,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":41,"recovering_keys_per_sec":0,"num_objects_recovered":500,"num_bytes_recovered":500,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:51:57.865 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:51:57.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:51:58.176 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:51:58.176 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:51:58.197 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2024-09-16T02:51:57.739716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19939,"num_bytes_recovered":137897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":352764,"kb_used_data":168220,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188390916,"statfs":{"total":193273528320,"available":192912297984,"internally_reserved":0,"allocated":172257280,"data_stored":490849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1000,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":6809,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:54.093800+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:54.093800+0000","last_peered":"2024-09-16T02:51:54.093800+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:54.093800+0000","last_undegraded":"2024-09-16T02:51:54.093800+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6268,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":903,"num_bytes_recovered":903,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":903},{"shards":"2,3","objects":3134}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":14248,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19939,"num_bytes_recovered":137897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352092160,"data_stored":701642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187652,"kb_used_data":84276,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94184188,"statfs":{"total":96636764160,"available":96444608512,"internally_reserved":0,"allocated":86298624,"data_stored":245466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673241,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165112,"kb_used_data":83944,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94206728,"statfs":{"total":96636764160,"available":96467689472,"internally_reserved":0,"allocated":85958656,"data_stored":245383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986229,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85741568,"data_stored":138847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86024192,"data_stored":138916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:01.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:01.560 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:01.561 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:01.583 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2024-09-16T02:51:59.740219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13722,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20202,"num_bytes_recovered":138160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":352764,"kb_used_data":168220,"kb_used_omap":3,"kb_used_meta":184508,"kb_avail":188390916,"statfs":{"total":193273528320,"available":192912297984,"internally_reserved":0,"allocated":172257280,"data_stored":490849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":188937098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1526,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003415"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":7598,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:59.110981+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:59.110981+0000","last_peered":"2024-09-16T02:51:59.110981+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:59.110981+0000","last_undegraded":"2024-09-16T02:51:59.110981+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5742,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1166,"num_bytes_recovered":1166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1166},{"shards":"2,3","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13722,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20202,"num_bytes_recovered":138160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352092160,"data_stored":701642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187652,"kb_used_data":84276,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94184188,"statfs":{"total":96636764160,"available":96444608512,"internally_reserved":0,"allocated":86298624,"data_stored":245466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673241,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165112,"kb_used_data":83944,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94206728,"statfs":{"total":96636764160,"available":96467689472,"internally_reserved":0,"allocated":85958656,"data_stored":245383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986229,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85741568,"data_stored":138847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86024192,"data_stored":138916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:01.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:01.901 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:01.902 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:01.922 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2024-09-16T02:52:01.740777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13722,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20202,"num_bytes_recovered":138160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":358000,"kb_used_data":169296,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188385680,"statfs":{"total":193273528320,"available":192906936320,"internally_reserved":0,"allocated":173359104,"data_stored":491118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1036,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":7598,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:59.110981+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:59.110981+0000","last_peered":"2024-09-16T02:51:59.110981+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:59.110981+0000","last_undegraded":"2024-09-16T02:51:59.110981+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5742,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1166,"num_bytes_recovered":1166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1166},{"shards":"2,3","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13722,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20202,"num_bytes_recovered":138160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":353259520,"data_stored":701927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187652,"kb_used_data":84276,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94184188,"statfs":{"total":96636764160,"available":96444608512,"internally_reserved":0,"allocated":86298624,"data_stored":245466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170348,"kb_used_data":85020,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94201492,"statfs":{"total":96636764160,"available":96462327808,"internally_reserved":0,"allocated":87060480,"data_stored":245652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986230,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86908928,"data_stored":139132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86024192,"data_stored":138916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:01.922 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:01.923 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:01.923 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:01.923 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:01.923 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:01.923 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 7598, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:51:59.110981+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:51:59.110981+0000', 'last_peered': '2024-09-16T02:51:59.110981+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:51:59.110981+0000', 'last_undegraded': '2024-09-16T02:51:59.110981+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5742, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1166, 'num_bytes_recovered': 1166, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1166}, {'shards': '2,3', 'objects': 2871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:01.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:02.357 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1381,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":366592000,"bytes_avail":192906936320,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":13722,"misplaced_total":63846,"misplaced_ratio":0.21492340945399868,"recovering_objects_per_sec":43,"recovering_bytes_per_sec":43,"recovering_keys_per_sec":0,"num_objects_recovered":518,"num_bytes_recovered":518,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:02.378 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:02.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:02.710 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:02.710 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:02.733 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2024-09-16T02:52:01.740777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13722,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20202,"num_bytes_recovered":138160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":358000,"kb_used_data":169296,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188385680,"statfs":{"total":193273528320,"available":192906936320,"internally_reserved":0,"allocated":173359104,"data_stored":491118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1036,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":518,"num_bytes_recovered":518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":7598,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:51:59.110981+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:51:59.110981+0000","last_peered":"2024-09-16T02:51:59.110981+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:51:59.110981+0000","last_undegraded":"2024-09-16T02:51:59.110981+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5742,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1166,"num_bytes_recovered":1166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1166},{"shards":"2,3","objects":2871}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13722,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20202,"num_bytes_recovered":138160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":353259520,"data_stored":701927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187652,"kb_used_data":84276,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94184188,"statfs":{"total":96636764160,"available":96444608512,"internally_reserved":0,"allocated":86298624,"data_stored":245466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170348,"kb_used_data":85020,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94201492,"statfs":{"total":96636764160,"available":96462327808,"internally_reserved":0,"allocated":87060480,"data_stored":245652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986230,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86908928,"data_stored":139132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86024192,"data_stored":138916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:05.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:06.097 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:06.098 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:06.119 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":726,"stamp":"2024-09-16T02:52:05.741882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13162,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20482,"num_bytes_recovered":138440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":359176,"kb_used_data":170472,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188384504,"statfs":{"total":193273528320,"available":192905732096,"internally_reserved":0,"allocated":174563328,"data_stored":491412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003195"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":8438,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:04.130755+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:04.130755+0000","last_peered":"2024-09-16T02:52:04.130755+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:04.130755+0000","last_undegraded":"2024-09-16T02:52:04.130755+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5182,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":1446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1446},{"shards":"2,3","objects":2591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13162,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20482,"num_bytes_recovered":138440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":354471936,"data_stored":702223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640538,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188828,"kb_used_data":85452,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94183012,"statfs":{"total":96636764160,"available":96443404288,"internally_reserved":0,"allocated":87502848,"data_stored":245760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170348,"kb_used_data":85020,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94201492,"statfs":{"total":96636764160,"available":96462327808,"internally_reserved":0,"allocated":87060480,"data_stored":245652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986230,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86908928,"data_stored":139132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87236608,"data_stored":139212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:06.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:06.467 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:06.486 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":726,"stamp":"2024-09-16T02:52:05.741882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13162,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20482,"num_bytes_recovered":138440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":359176,"kb_used_data":170472,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188384504,"statfs":{"total":193273528320,"available":192905732096,"internally_reserved":0,"allocated":174563328,"data_stored":491412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003195"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":8438,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:04.130755+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:04.130755+0000","last_peered":"2024-09-16T02:52:04.130755+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:04.130755+0000","last_undegraded":"2024-09-16T02:52:04.130755+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5182,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":1446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1446},{"shards":"2,3","objects":2591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13162,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20482,"num_bytes_recovered":138440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":354471936,"data_stored":702223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640538,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188828,"kb_used_data":85452,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94183012,"statfs":{"total":96636764160,"available":96443404288,"internally_reserved":0,"allocated":87502848,"data_stored":245760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170348,"kb_used_data":85020,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94201492,"statfs":{"total":96636764160,"available":96462327808,"internally_reserved":0,"allocated":87060480,"data_stored":245652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986230,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86908928,"data_stored":139132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87236608,"data_stored":139212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:06.487 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:06.487 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:06.487 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:06.487 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:06.487 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:06.487 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 8438, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:04.130755+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:52:04.130755+0000', 'last_peered': '2024-09-16T02:52:04.130755+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:52:04.130755+0000', 'last_undegraded': '2024-09-16T02:52:04.130755+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5182, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1446, 'num_bytes_recovered': 1446, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1446}, {'shards': '2,3', 'objects': 2591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:06.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:06.854 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:06.855 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1386,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":367796224,"bytes_avail":192905732096,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":13162,"misplaced_total":63846,"misplaced_ratio":0.20615230398145537,"recovering_objects_per_sec":66,"recovering_bytes_per_sec":66,"recovering_keys_per_sec":0,"num_objects_recovered":798,"num_bytes_recovered":798,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:06.873 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:06.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:07.215 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:07.215 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:07.234 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":726,"stamp":"2024-09-16T02:52:05.741882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13162,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20482,"num_bytes_recovered":138440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":359176,"kb_used_data":170472,"kb_used_omap":3,"kb_used_meta":188668,"kb_avail":188384504,"statfs":{"total":193273528320,"available":192905732096,"internally_reserved":0,"allocated":174563328,"data_stored":491412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":193196938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003195"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":8438,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:04.130755+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:04.130755+0000","last_peered":"2024-09-16T02:52:04.130755+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:04.130755+0000","last_undegraded":"2024-09-16T02:52:04.130755+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5182,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":1446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1446},{"shards":"2,3","objects":2591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":13162,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20482,"num_bytes_recovered":138440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":354471936,"data_stored":702223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640538,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188828,"kb_used_data":85452,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94183012,"statfs":{"total":96636764160,"available":96443404288,"internally_reserved":0,"allocated":87502848,"data_stored":245760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":170348,"kb_used_data":85020,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94201492,"statfs":{"total":96636764160,"available":96462327808,"internally_reserved":0,"allocated":87060480,"data_stored":245652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986230,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86908928,"data_stored":139132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87236608,"data_stored":139212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:10.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:10.583 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:10.583 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:10.602 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2024-09-16T02:52:09.743123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":365692,"kb_used_data":172828,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188377988,"statfs":{"total":193273528320,"available":192899059712,"internally_reserved":0,"allocated":176975872,"data_stored":492001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1676,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":9324,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:09.130841+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:09.130841+0000","last_peered":"2024-09-16T02:52:09.130841+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:09.130841+0000","last_undegraded":"2024-09-16T02:52:09.130841+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4592,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":1741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1741},{"shards":"2,3","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356925440,"data_stored":702822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640539,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194240,"kb_used_data":86704,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94177600,"statfs":{"total":96636764160,"available":96437862400,"internally_reserved":0,"allocated":88784896,"data_stored":246073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673243,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171452,"kb_used_data":86124,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94200388,"statfs":{"total":96636764160,"available":96461197312,"internally_reserved":0,"allocated":88190976,"data_stored":245928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":88133632,"data_stored":139431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88465408,"data_stored":139512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:10.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:10.927 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:10.948 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2024-09-16T02:52:09.743123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":365692,"kb_used_data":172828,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188377988,"statfs":{"total":193273528320,"available":192899059712,"internally_reserved":0,"allocated":176975872,"data_stored":492001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1676,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":9324,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:09.130841+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:09.130841+0000","last_peered":"2024-09-16T02:52:09.130841+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:09.130841+0000","last_undegraded":"2024-09-16T02:52:09.130841+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4592,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":1741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1741},{"shards":"2,3","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356925440,"data_stored":702822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640539,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194240,"kb_used_data":86704,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94177600,"statfs":{"total":96636764160,"available":96437862400,"internally_reserved":0,"allocated":88784896,"data_stored":246073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673243,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171452,"kb_used_data":86124,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94200388,"statfs":{"total":96636764160,"available":96461197312,"internally_reserved":0,"allocated":88190976,"data_stored":245928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":88133632,"data_stored":139431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88465408,"data_stored":139512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:10.949 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:10.949 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:10.949 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:10.949 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:10.949 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:10.949 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 9324, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:09.130841+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:52:09.130841+0000', 'last_peered': '2024-09-16T02:52:09.130841+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:52:09.130841+0000', 'last_undegraded': '2024-09-16T02:52:09.130841+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4592, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1741, 'num_bytes_recovered': 1741, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1741}, {'shards': '2,3', 'objects': 2296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:10.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:11.351 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:11.351 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1390,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":374468608,"bytes_avail":192899059712,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":12572,"misplaced_total":63846,"misplaced_ratio":0.19691131785859725,"recovering_objects_per_sec":69,"recovering_bytes_per_sec":69,"recovering_keys_per_sec":0,"num_objects_recovered":838,"num_bytes_recovered":838,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:11.372 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:11.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:11.697 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:11.716 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2024-09-16T02:52:09.743123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":365692,"kb_used_data":172828,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188377988,"statfs":{"total":193273528320,"available":192899059712,"internally_reserved":0,"allocated":176975872,"data_stored":492001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1676,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":9324,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:09.130841+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:09.130841+0000","last_peered":"2024-09-16T02:52:09.130841+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:09.130841+0000","last_undegraded":"2024-09-16T02:52:09.130841+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4592,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":1741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1741},{"shards":"2,3","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356925440,"data_stored":702822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640539,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194240,"kb_used_data":86704,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94177600,"statfs":{"total":96636764160,"available":96437862400,"internally_reserved":0,"allocated":88784896,"data_stored":246073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673243,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":171452,"kb_used_data":86124,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94200388,"statfs":{"total":96636764160,"available":96461197312,"internally_reserved":0,"allocated":88190976,"data_stored":245928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":88133632,"data_stored":139431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88465408,"data_stored":139512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:14.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:15.063 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:15.063 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:15.083 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2024-09-16T02:52:13.744037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":368264,"kb_used_data":175400,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188375416,"statfs":{"total":193273528320,"available":192896425984,"internally_reserved":0,"allocated":179609600,"data_stored":492644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1150,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":575,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":9324,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:09.130841+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:09.130841+0000","last_peered":"2024-09-16T02:52:09.130841+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:09.130841+0000","last_undegraded":"2024-09-16T02:52:09.130841+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4592,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":1741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1741},{"shards":"2,3","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":359641088,"data_stored":703485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640540,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195532,"kb_used_data":87996,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94176308,"statfs":{"total":96636764160,"available":96436539392,"internally_reserved":0,"allocated":90107904,"data_stored":246396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673244,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172732,"kb_used_data":87404,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94199108,"statfs":{"total":96636764160,"available":96459886592,"internally_reserved":0,"allocated":89501696,"data_stored":246248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986232,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89522176,"data_stored":139770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89792512,"data_stored":139836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:15.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:15.406 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:15.406 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:15.426 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2024-09-16T02:52:13.744037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":368264,"kb_used_data":175400,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188375416,"statfs":{"total":193273528320,"available":192896425984,"internally_reserved":0,"allocated":179609600,"data_stored":492644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1150,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":575,"num_bytes_recovered":575,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":9324,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:09.130841+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:09.130841+0000","last_peered":"2024-09-16T02:52:09.130841+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:09.130841+0000","last_undegraded":"2024-09-16T02:52:09.130841+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4592,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1741,"num_bytes_recovered":1741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1741},{"shards":"2,3","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":12572,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20777,"num_bytes_recovered":138735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":359641088,"data_stored":703485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640540,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195532,"kb_used_data":87996,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94176308,"statfs":{"total":96636764160,"available":96436539392,"internally_reserved":0,"allocated":90107904,"data_stored":246396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673244,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172732,"kb_used_data":87404,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94199108,"statfs":{"total":96636764160,"available":96459886592,"internally_reserved":0,"allocated":89501696,"data_stored":246248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986232,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89522176,"data_stored":139770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89792512,"data_stored":139836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:15.426 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:15.427 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:15.427 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:15.427 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:15.427 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:15.427 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 9324, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:09.130841+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:52:09.130841+0000', 'last_peered': '2024-09-16T02:52:09.130841+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:52:09.130841+0000', 'last_undegraded': '2024-09-16T02:52:09.130841+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4592, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1741, 'num_bytes_recovered': 1741, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1741}, {'shards': '2,3', 'objects': 2296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:15.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:15.862 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1395,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":377102336,"bytes_avail":192896425984,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":12572,"misplaced_total":63846,"misplaced_ratio":0.19691131785859725,"recovering_objects_per_sec":47,"recovering_bytes_per_sec":47,"recovering_keys_per_sec":0,"num_objects_recovered":575,"num_bytes_recovered":575,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:15.884 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:15.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:16.219 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:16.219 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:16.240 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":731,"stamp":"2024-09-16T02:52:15.744607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11910,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21108,"num_bytes_recovered":139066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":368264,"kb_used_data":175400,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188375416,"statfs":{"total":193273528320,"available":192896425984,"internally_reserved":0,"allocated":179609600,"data_stored":492644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1812,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003428"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":10316,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:14.134332+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:14.134332+0000","last_peered":"2024-09-16T02:52:14.134332+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:14.134332+0000","last_undegraded":"2024-09-16T02:52:14.134332+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3930,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2072,"num_bytes_recovered":2072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2072},{"shards":"2,3","objects":1965}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11910,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21108,"num_bytes_recovered":139066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":359641088,"data_stored":703485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640540,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195532,"kb_used_data":87996,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94176308,"statfs":{"total":96636764160,"available":96436539392,"internally_reserved":0,"allocated":90107904,"data_stored":246396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673244,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":172732,"kb_used_data":87404,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":94199108,"statfs":{"total":96636764160,"available":96459886592,"internally_reserved":0,"allocated":89501696,"data_stored":246248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986232,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":89522176,"data_stored":139770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":89792512,"data_stored":139836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:19.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:19.599 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:19.619 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2024-09-16T02:52:17.745105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11910,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21108,"num_bytes_recovered":139066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":375256,"kb_used_data":178232,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188368424,"statfs":{"total":193273528320,"available":192889266176,"internally_reserved":0,"allocated":182509568,"data_stored":493352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1252,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":626,"num_bytes_recovered":626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003223"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":10316,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:14.134332+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:14.134332+0000","last_peered":"2024-09-16T02:52:14.134332+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:14.134332+0000","last_undegraded":"2024-09-16T02:52:14.134332+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3930,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2072,"num_bytes_recovered":2072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2072},{"shards":"2,3","objects":1965}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11910,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21108,"num_bytes_recovered":139066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362524672,"data_stored":704189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640541,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196904,"kb_used_data":89368,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94174936,"statfs":{"total":96636764160,"available":96435134464,"internally_reserved":0,"allocated":91512832,"data_stored":246739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178352,"kb_used_data":88864,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94193488,"statfs":{"total":96636764160,"available":96454131712,"internally_reserved":0,"allocated":90996736,"data_stored":246613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90984448,"data_stored":140127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91213824,"data_stored":140183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:19.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:19.948 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:19.968 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2024-09-16T02:52:19.745702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":375256,"kb_used_data":178232,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188368424,"statfs":{"total":193273528320,"available":192889266176,"internally_reserved":0,"allocated":182509568,"data_stored":493352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1968,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003192"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":11390,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:19.136502+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:19.136502+0000","last_peered":"2024-09-16T02:52:19.136502+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:19.136502+0000","last_undegraded":"2024-09-16T02:52:19.136502+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3214,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2430,"num_bytes_recovered":2430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2430},{"shards":"2,3","objects":1607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362524672,"data_stored":704189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640541,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196904,"kb_used_data":89368,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94174936,"statfs":{"total":96636764160,"available":96435134464,"internally_reserved":0,"allocated":91512832,"data_stored":246739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178352,"kb_used_data":88864,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94193488,"statfs":{"total":96636764160,"available":96454131712,"internally_reserved":0,"allocated":90996736,"data_stored":246613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90984448,"data_stored":140127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91213824,"data_stored":140183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:19.969 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:19.969 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:19.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:19.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:19.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:19.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 11390, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:19.136502+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:52:19.136502+0000', 'last_peered': '2024-09-16T02:52:19.136502+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:52:19.136502+0000', 'last_undegraded': '2024-09-16T02:52:19.136502+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3214, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2430, 'num_bytes_recovered': 2430, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2430}, {'shards': '2,3', 'objects': 1607}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:19.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:20.371 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:20.372 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1399,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":384262144,"bytes_avail":192889266176,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":11194,"misplaced_total":63846,"misplaced_ratio":0.17532813332080319,"recovering_objects_per_sec":81,"recovering_bytes_per_sec":81,"recovering_keys_per_sec":0,"num_objects_recovered":984,"num_bytes_recovered":984,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:20.394 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:20.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:20.722 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:20.723 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:20.741 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2024-09-16T02:52:19.745702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":375256,"kb_used_data":178232,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188368424,"statfs":{"total":193273528320,"available":192889266176,"internally_reserved":0,"allocated":182509568,"data_stored":493352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1968,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003192"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":11390,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:19.136502+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:19.136502+0000","last_peered":"2024-09-16T02:52:19.136502+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:19.136502+0000","last_undegraded":"2024-09-16T02:52:19.136502+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3214,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2430,"num_bytes_recovered":2430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2430},{"shards":"2,3","objects":1607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362524672,"data_stored":704189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640541,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196904,"kb_used_data":89368,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94174936,"statfs":{"total":96636764160,"available":96435134464,"internally_reserved":0,"allocated":91512832,"data_stored":246739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178352,"kb_used_data":88864,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94193488,"statfs":{"total":96636764160,"available":96454131712,"internally_reserved":0,"allocated":90996736,"data_stored":246613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90984448,"data_stored":140127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91213824,"data_stored":140183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:23.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:24.093 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:24.093 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:24.111 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2024-09-16T02:52:23.746707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":382764,"kb_used_data":181576,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188360916,"statfs":{"total":193273528320,"available":192881577984,"internally_reserved":0,"allocated":185933824,"data_stored":494188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1378,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":11390,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:19.136502+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:19.136502+0000","last_peered":"2024-09-16T02:52:19.136502+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:19.136502+0000","last_undegraded":"2024-09-16T02:52:19.136502+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3214,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2430,"num_bytes_recovered":2430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2430},{"shards":"2,3","objects":1607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365924352,"data_stored":705019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640542,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":202752,"kb_used_data":91056,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94169088,"statfs":{"total":96636764160,"available":96429146112,"internally_reserved":0,"allocated":93241344,"data_stored":247161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673246,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180012,"kb_used_data":90520,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94191828,"statfs":{"total":96636764160,"available":96452431872,"internally_reserved":0,"allocated":92692480,"data_stored":247027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986234,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":92663808,"data_stored":140537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92934144,"data_stored":140603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:24.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:24.429 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:24.429 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:24.451 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2024-09-16T02:52:23.746707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":382764,"kb_used_data":181576,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188360916,"statfs":{"total":193273528320,"available":192881577984,"internally_reserved":0,"allocated":185933824,"data_stored":494188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1378,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":11390,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:19.136502+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:19.136502+0000","last_peered":"2024-09-16T02:52:19.136502+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:19.136502+0000","last_undegraded":"2024-09-16T02:52:19.136502+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3214,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2430,"num_bytes_recovered":2430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2430},{"shards":"2,3","objects":1607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365924352,"data_stored":705019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640542,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":202752,"kb_used_data":91056,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94169088,"statfs":{"total":96636764160,"available":96429146112,"internally_reserved":0,"allocated":93241344,"data_stored":247161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673246,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180012,"kb_used_data":90520,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94191828,"statfs":{"total":96636764160,"available":96452431872,"internally_reserved":0,"allocated":92692480,"data_stored":247027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986234,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":92663808,"data_stored":140537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92934144,"data_stored":140603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:24.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:24.452 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:24.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:24.453 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:24.453 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:24.453 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 11390, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:19.136502+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:52:19.136502+0000', 'last_peered': '2024-09-16T02:52:19.136502+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:52:19.136502+0000', 'last_undegraded': '2024-09-16T02:52:19.136502+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3214, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2430, 'num_bytes_recovered': 2430, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2430}, {'shards': '2,3', 'objects': 1607}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:24.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:24.871 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:24.872 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1404,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":391950336,"bytes_avail":192881577984,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":11194,"misplaced_total":63846,"misplaced_ratio":0.17532813332080319,"recovering_objects_per_sec":57,"recovering_bytes_per_sec":57,"recovering_keys_per_sec":0,"num_objects_recovered":689,"num_bytes_recovered":689,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:24.893 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:24.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:25.232 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:25.232 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:25.253 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2024-09-16T02:52:23.746707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":382764,"kb_used_data":181576,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188360916,"statfs":{"total":193273528320,"available":192881577984,"internally_reserved":0,"allocated":185933824,"data_stored":494188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1378,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":11390,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:19.136502+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:19.136502+0000","last_peered":"2024-09-16T02:52:19.136502+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:19.136502+0000","last_undegraded":"2024-09-16T02:52:19.136502+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3214,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2430,"num_bytes_recovered":2430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2430},{"shards":"2,3","objects":1607}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":11194,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21466,"num_bytes_recovered":139424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365924352,"data_stored":705019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640542,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":202752,"kb_used_data":91056,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94169088,"statfs":{"total":96636764160,"available":96429146112,"internally_reserved":0,"allocated":93241344,"data_stored":247161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673246,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":180012,"kb_used_data":90520,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94191828,"statfs":{"total":96636764160,"available":96452431872,"internally_reserved":0,"allocated":92692480,"data_stored":247027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986234,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":92663808,"data_stored":140537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92934144,"data_stored":140603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:28.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:28.620 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:28.620 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:28.641 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-09-16T02:52:27.748018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":10294,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21916,"num_bytes_recovered":139874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":386328,"kb_used_data":185144,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188357352,"statfs":{"total":193273528320,"available":192877928448,"internally_reserved":0,"allocated":189587456,"data_stored":495080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":12741,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:24.077825+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:24.077825+0000","last_peered":"2024-09-16T02:52:24.077825+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:24.077825+0000","last_undegraded":"2024-09-16T02:52:24.077825+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2314,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2880,"num_bytes_recovered":2880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2880},{"shards":"2,3","objects":1157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":10294,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21916,"num_bytes_recovered":139874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":369532928,"data_stored":705900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640543,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204536,"kb_used_data":92840,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94167304,"statfs":{"total":96636764160,"available":96427319296,"internally_reserved":0,"allocated":95068160,"data_stored":247607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673247,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181792,"kb_used_data":92304,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94190048,"statfs":{"total":96636764160,"available":96450609152,"internally_reserved":0,"allocated":94519296,"data_stored":247473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986235,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94433280,"data_stored":140969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94773248,"data_stored":141052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:28.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:28.983 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:28.983 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:29.003 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-09-16T02:52:27.748018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":10294,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21916,"num_bytes_recovered":139874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":386328,"kb_used_data":185144,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188357352,"statfs":{"total":193273528320,"available":192877928448,"internally_reserved":0,"allocated":189587456,"data_stored":495080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":12741,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:24.077825+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:24.077825+0000","last_peered":"2024-09-16T02:52:24.077825+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:24.077825+0000","last_undegraded":"2024-09-16T02:52:24.077825+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2314,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2880,"num_bytes_recovered":2880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2880},{"shards":"2,3","objects":1157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":10294,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21916,"num_bytes_recovered":139874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":369532928,"data_stored":705900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640543,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204536,"kb_used_data":92840,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94167304,"statfs":{"total":96636764160,"available":96427319296,"internally_reserved":0,"allocated":95068160,"data_stored":247607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673247,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181792,"kb_used_data":92304,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94190048,"statfs":{"total":96636764160,"available":96450609152,"internally_reserved":0,"allocated":94519296,"data_stored":247473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986235,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94433280,"data_stored":140969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94773248,"data_stored":141052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:29.004 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:29.004 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:29.004 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:29.004 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:29.004 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:29.004 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 12741, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:24.077825+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:52:24.077825+0000', 'last_peered': '2024-09-16T02:52:24.077825+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:52:24.077825+0000', 'last_undegraded': '2024-09-16T02:52:24.077825+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2314, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2880, 'num_bytes_recovered': 2880, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2880}, {'shards': '2,3', 'objects': 1157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:29.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:29.418 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1409,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":395599872,"bytes_avail":192877928448,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":10294,"misplaced_total":63846,"misplaced_ratio":0.16123171381135859,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":67,"recovering_keys_per_sec":0,"num_objects_recovered":808,"num_bytes_recovered":808,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:29.440 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:29.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:29.780 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:29.780 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:29.799 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":738,"stamp":"2024-09-16T02:52:29.748683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":386328,"kb_used_data":185144,"kb_used_omap":3,"kb_used_meta":201148,"kb_avail":188357352,"statfs":{"total":193273528320,"available":192877928448,"internally_reserved":0,"allocated":189587456,"data_stored":495080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":205976458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2708,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":1354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003577"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":14378,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:29.150494+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:29.150494+0000","last_peered":"2024-09-16T02:52:29.150494+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:29.150494+0000","last_undegraded":"2024-09-16T02:52:29.150494+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1222,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":3426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3426},{"shards":"2,3","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":369532928,"data_stored":705900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640543,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204536,"kb_used_data":92840,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":94167304,"statfs":{"total":96636764160,"available":96427319296,"internally_reserved":0,"allocated":95068160,"data_stored":247607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673247,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":181792,"kb_used_data":92304,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":94190048,"statfs":{"total":96636764160,"available":96450609152,"internally_reserved":0,"allocated":94519296,"data_stored":247473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986235,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94433280,"data_stored":140969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94773248,"data_stored":141052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:32.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:33.162 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:33.162 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:33.182 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2024-09-16T02:52:33.005928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":383576,"kb_used_data":190712,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188360104,"statfs":{"total":193273528320,"available":192880746496,"internally_reserved":0,"allocated":195289088,"data_stored":496472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1992,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":996,"num_bytes_recovered":996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.259818"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":14378,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:29.150494+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:29.150494+0000","last_peered":"2024-09-16T02:52:29.150494+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:29.150494+0000","last_undegraded":"2024-09-16T02:52:29.150494+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1222,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":3426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3426},{"shards":"2,3","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375394304,"data_stored":707331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640544,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195008,"kb_used_data":95792,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94176832,"statfs":{"total":96636764160,"available":96437075968,"internally_reserved":0,"allocated":98091008,"data_stored":248345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673248,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188568,"kb_used_data":94920,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94183272,"statfs":{"total":96636764160,"available":96443670528,"internally_reserved":0,"allocated":97198080,"data_stored":248127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986236,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97234944,"data_stored":141653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97832960,"data_stored":141799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:33.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:33.527 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:33.528 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:33.547 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2024-09-16T02:52:33.005928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":383576,"kb_used_data":190712,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188360104,"statfs":{"total":193273528320,"available":192880746496,"internally_reserved":0,"allocated":195289088,"data_stored":496472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1992,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":996,"num_bytes_recovered":996,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.259818"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":14378,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:29.150494+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:29.150494+0000","last_peered":"2024-09-16T02:52:29.150494+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:29.150494+0000","last_undegraded":"2024-09-16T02:52:29.150494+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1222,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":3426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3426},{"shards":"2,3","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375394304,"data_stored":707331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640544,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195008,"kb_used_data":95792,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94176832,"statfs":{"total":96636764160,"available":96437075968,"internally_reserved":0,"allocated":98091008,"data_stored":248345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673248,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188568,"kb_used_data":94920,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94183272,"statfs":{"total":96636764160,"available":96443670528,"internally_reserved":0,"allocated":97198080,"data_stored":248127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986236,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97234944,"data_stored":141653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97832960,"data_stored":141799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:33.548 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:33.548 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4038, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373983+0000', 'last_change': '2024-09-16T02:47:26.528258+0000', 'last_active': '2024-09-16T02:51:36.373983+0000', 'last_peered': '2024-09-16T02:51:36.373983+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:51:36.373983+0000', 'last_undegraded': '2024-09-16T02:47:26.513795+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3956, 'num_objects_misplaced': 3956, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 3956}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:33.548 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:33.548 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4095, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:51:36.373967+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:51:36.373967+0000', 'last_peered': '2024-09-16T02:51:36.373967+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:51:36.373967+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:33.548 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-16T02:52:33.548 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "19'4037", 'reported_seq': 14378, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:29.150494+0000', 'last_change': '2024-09-16T02:51:35.318189+0000', 'last_active': '2024-09-16T02:52:29.150494+0000', 'last_peered': '2024-09-16T02:52:29.150494+0000', 'last_clean': '2024-09-16T02:46:37.653464+0000', 'last_became_active': '2024-09-16T02:47:26.515758+0000', 'last_became_peered': '2024-09-16T02:47:26.515758+0000', 'last_unstale': '2024-09-16T02:52:29.150494+0000', 'last_undegraded': '2024-09-16T02:52:29.150494+0000', 'last_fullsized': '2024-09-16T02:47:26.333595+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4037, 'log_dups_size': 0, 'ondisk_log_size': 4037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56437, 'num_objects': 4037, 'num_object_clones': 0, 'num_object_copies': 8074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1222, 'num_objects_unfound': 0, 'num_objects_dirty': 4037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4037, 'num_write_kb': 4085, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3426, 'num_bytes_recovered': 3426, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3426}, {'shards': '2,3', 'objects': 611}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:33.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:33.935 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:33.935 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7980/63846 objects degraded (12.499%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1413,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":402538496,"bytes_avail":192870989824,"bytes_total":193273528320,"degraded_objects":7980,"degraded_total":63846,"degraded_ratio":0.12498825298374212,"misplaced_objects":9202,"misplaced_total":63846,"misplaced_ratio":0.14412805813989912,"recovering_objects_per_sec":82,"recovering_bytes_per_sec":82,"recovering_keys_per_sec":0,"num_objects_recovered":996,"num_bytes_recovered":996,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:33.955 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:33.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:34.309 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:34.310 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:34.336 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2024-09-16T02:52:34.007146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":383576,"kb_used_data":190712,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188360104,"statfs":{"total":193273528320,"available":192880746496,"internally_reserved":0,"allocated":195289088,"data_stored":496472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1092,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":546,"num_bytes_recovered":546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.259668"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4038,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373983+0000","last_change":"2024-09-16T02:47:26.528258+0000","last_active":"2024-09-16T02:51:36.373983+0000","last_peered":"2024-09-16T02:51:36.373983+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:51:36.373983+0000","last_undegraded":"2024-09-16T02:47:26.513795+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4095,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:51:36.373967+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:51:36.373967+0000","last_peered":"2024-09-16T02:51:36.373967+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:51:36.373967+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":14378,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:29.150494+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:29.150494+0000","last_peered":"2024-09-16T02:52:29.150494+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:29.150494+0000","last_undegraded":"2024-09-16T02:52:29.150494+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1222,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":3426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3426},{"shards":"2,3","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7980,"num_objects_misplaced":9202,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22462,"num_bytes_recovered":140420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375394304,"data_stored":707331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640544,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195008,"kb_used_data":95792,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94176832,"statfs":{"total":96636764160,"available":96437075968,"internally_reserved":0,"allocated":98091008,"data_stored":248345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673248,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188568,"kb_used_data":94920,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94183272,"statfs":{"total":96636764160,"available":96443670528,"internally_reserved":0,"allocated":97198080,"data_stored":248127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986236,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97234944,"data_stored":141653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97832960,"data_stored":141799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:37.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:37.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:37.690 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:37.709 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":743,"stamp":"2024-09-16T02:52:35.750328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":12984,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22549,"num_bytes_recovered":140507,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":383576,"kb_used_data":190712,"kb_used_omap":3,"kb_used_meta":192828,"kb_avail":188360104,"statfs":{"total":193273528320,"available":192880746496,"internally_reserved":0,"allocated":195289088,"data_stored":496472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":197456778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":2690,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002310"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":3999,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.374838+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:51:36.374838+0000","last_peered":"2024-09-16T02:51:36.374838+0000","last_clean":"2024-09-16T02:51:36.374838+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:51:36.374838+0000","last_undegraded":"2024-09-16T02:51:36.374838+0000","last_fullsized":"2024-09-16T02:51:36.374838+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4056,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380316+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:51:36.380316+0000","last_peered":"2024-09-16T02:51:36.380316+0000","last_clean":"2024-09-16T02:51:36.380316+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:51:36.380316+0000","last_undegraded":"2024-09-16T02:51:36.380316+0000","last_fullsized":"2024-09-16T02:51:36.380316+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4115,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.391142+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:51:36.391142+0000","last_peered":"2024-09-16T02:51:36.391142+0000","last_clean":"2024-09-16T02:51:36.391142+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:51:36.391142+0000","last_undegraded":"2024-09-16T02:51:36.391142+0000","last_fullsized":"2024-09-16T02:51:36.391142+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4304,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:34.144119+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:34.144119+0000","last_peered":"2024-09-16T02:52:34.144119+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:34.144119+0000","last_undegraded":"2024-09-16T02:52:34.144119+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7738,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":87,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":87},{"shards":"2,3","objects":3869}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4044,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380006+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:51:36.380006+0000","last_peered":"2024-09-16T02:51:36.380006+0000","last_clean":"2024-09-16T02:51:36.380006+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:51:36.380006+0000","last_undegraded":"2024-09-16T02:51:36.380006+0000","last_fullsized":"2024-09-16T02:51:36.380006+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3986,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T02:51:36.380385+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:51:36.380385+0000","last_peered":"2024-09-16T02:51:36.380385+0000","last_clean":"2024-09-16T02:51:36.380385+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:51:36.380385+0000","last_undegraded":"2024-09-16T02:51:36.380385+0000","last_fullsized":"2024-09-16T02:51:36.380385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":14378,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:29.150494+0000","last_change":"2024-09-16T02:51:35.318189+0000","last_active":"2024-09-16T02:52:29.150494+0000","last_peered":"2024-09-16T02:52:29.150494+0000","last_clean":"2024-09-16T02:46:37.653464+0000","last_became_active":"2024-09-16T02:47:26.515758+0000","last_became_peered":"2024-09-16T02:47:26.515758+0000","last_unstale":"2024-09-16T02:52:29.150494+0000","last_undegraded":"2024-09-16T02:52:29.150494+0000","last_fullsized":"2024-09-16T02:47:26.333595+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1222,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":3426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3426},{"shards":"2,3","objects":611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":12984,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22549,"num_bytes_recovered":140507,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375394304,"data_stored":707331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640544,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195008,"kb_used_data":95792,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94176832,"statfs":{"total":96636764160,"available":96437075968,"internally_reserved":0,"allocated":98091008,"data_stored":248345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673248,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188568,"kb_used_data":94920,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":94183272,"statfs":{"total":96636764160,"available":96443670528,"internally_reserved":0,"allocated":97198080,"data_stored":248127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986236,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97234944,"data_stored":141653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97832960,"data_stored":141799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49340416,"data_stored":143016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:37.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:38.069 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:38.069 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:38.091 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2024-09-16T02:52:37.750892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11762,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23160,"num_bytes_recovered":193518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":391208,"kb_used_data":194184,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188352472,"statfs":{"total":193273528320,"available":192872931328,"internally_reserved":0,"allocated":198844416,"data_stored":612886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":2560,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":698,"num_bytes_recovered":53098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002208"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4304,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:34.144119+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:34.144119+0000","last_peered":"2024-09-16T02:52:34.144119+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:34.144119+0000","last_undegraded":"2024-09-16T02:52:34.144119+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7738,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":87,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":87},{"shards":"2,3","objects":3869}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4048,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017513+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017513+0000","last_peered":"2024-09-16T02:52:34.017513+0000","last_clean":"2024-09-16T02:52:34.017513+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017513+0000","last_undegraded":"2024-09-16T02:52:34.017513+0000","last_fullsized":"2024-09-16T02:52:34.017513+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11762,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23160,"num_bytes_recovered":193518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362012672,"data_stored":756452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196432,"kb_used_data":97216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94175408,"statfs":{"total":96636764160,"available":96435617792,"internally_reserved":0,"allocated":99549184,"data_stored":306474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673249,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194776,"kb_used_data":96968,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":94177064,"statfs":{"total":96636764160,"available":96437313536,"internally_reserved":0,"allocated":99295232,"data_stored":306412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986237,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99041280,"data_stored":194482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99229696,"data_stored":194528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:38.092 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:38.092 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 4304, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:34.144119+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:52:34.144119+0000', 'last_peered': '2024-09-16T02:52:34.144119+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:52:34.144119+0000', 'last_undegraded': '2024-09-16T02:52:34.144119+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7738, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 87, 'num_bytes_recovered': 87, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 87}, {'shards': '2,3', 'objects': 3869}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:38.092 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:38.092 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:38.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:38.504 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:38.505 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1418,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":400596992,"bytes_avail":192872931328,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":11762,"misplaced_total":63846,"misplaced_ratio":0.18422454030009711,"recovering_objects_per_sec":87,"recovering_bytes_per_sec":6635,"recovering_keys_per_sec":0,"num_objects_recovered":698,"num_bytes_recovered":53098,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:38.525 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:38.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:38.859 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:38.860 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:38.880 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2024-09-16T02:52:37.750892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11762,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23160,"num_bytes_recovered":193518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":391208,"kb_used_data":194184,"kb_used_omap":3,"kb_used_meta":196988,"kb_avail":188352472,"statfs":{"total":193273528320,"available":192872931328,"internally_reserved":0,"allocated":198844416,"data_stored":612886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":201716618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":2560,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":698,"num_bytes_recovered":53098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002208"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4304,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:34.144119+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:34.144119+0000","last_peered":"2024-09-16T02:52:34.144119+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:34.144119+0000","last_undegraded":"2024-09-16T02:52:34.144119+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7738,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":87,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":87},{"shards":"2,3","objects":3869}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4048,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017513+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017513+0000","last_peered":"2024-09-16T02:52:34.017513+0000","last_clean":"2024-09-16T02:52:34.017513+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017513+0000","last_undegraded":"2024-09-16T02:52:34.017513+0000","last_fullsized":"2024-09-16T02:52:34.017513+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11762,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23160,"num_bytes_recovered":193518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362012672,"data_stored":756452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196432,"kb_used_data":97216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94175408,"statfs":{"total":96636764160,"available":96435617792,"internally_reserved":0,"allocated":99549184,"data_stored":306474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673249,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":194776,"kb_used_data":96968,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":94177064,"statfs":{"total":96636764160,"available":96437313536,"internally_reserved":0,"allocated":99295232,"data_stored":306412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986237,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99041280,"data_stored":194482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99229696,"data_stored":194528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:41.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:42.245 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:42.265 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2024-09-16T02:52:41.751839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11236,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23423,"num_bytes_recovered":193781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":375516,"kb_used_data":195132,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188368164,"statfs":{"total":193273528320,"available":192888999936,"internally_reserved":0,"allocated":199815168,"data_stored":613123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":53361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.745911"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":5093,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:39.132903+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:39.132903+0000","last_peered":"2024-09-16T02:52:39.132903+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:39.132903+0000","last_undegraded":"2024-09-16T02:52:39.132903+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7212,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":350},{"shards":"2,3","objects":3606}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4048,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017513+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017513+0000","last_peered":"2024-09-16T02:52:34.017513+0000","last_clean":"2024-09-16T02:52:34.017513+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017513+0000","last_undegraded":"2024-09-16T02:52:34.017513+0000","last_fullsized":"2024-09-16T02:52:34.017513+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11236,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23423,"num_bytes_recovered":193781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362987520,"data_stored":756690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196432,"kb_used_data":97216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94175408,"statfs":{"total":96636764160,"available":96435617792,"internally_reserved":0,"allocated":99549184,"data_stored":306474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179084,"kb_used_data":97916,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94192756,"statfs":{"total":96636764160,"available":96453382144,"internally_reserved":0,"allocated":100265984,"data_stored":306649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100016128,"data_stored":194720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99229696,"data_stored":194528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:42.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:42.602 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:42.602 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:42.624 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2024-09-16T02:52:41.751839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11236,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23423,"num_bytes_recovered":193781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":375516,"kb_used_data":195132,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188368164,"statfs":{"total":193273528320,"available":192888999936,"internally_reserved":0,"allocated":199815168,"data_stored":613123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":53361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.745911"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":5093,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:39.132903+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:39.132903+0000","last_peered":"2024-09-16T02:52:39.132903+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:39.132903+0000","last_undegraded":"2024-09-16T02:52:39.132903+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7212,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":350},{"shards":"2,3","objects":3606}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4048,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017513+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017513+0000","last_peered":"2024-09-16T02:52:34.017513+0000","last_clean":"2024-09-16T02:52:34.017513+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017513+0000","last_undegraded":"2024-09-16T02:52:34.017513+0000","last_fullsized":"2024-09-16T02:52:34.017513+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11236,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23423,"num_bytes_recovered":193781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362987520,"data_stored":756690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196432,"kb_used_data":97216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94175408,"statfs":{"total":96636764160,"available":96435617792,"internally_reserved":0,"allocated":99549184,"data_stored":306474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179084,"kb_used_data":97916,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94192756,"statfs":{"total":96636764160,"available":96453382144,"internally_reserved":0,"allocated":100265984,"data_stored":306649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100016128,"data_stored":194720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99229696,"data_stored":194528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:42.625 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:42.625 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 5093, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:39.132903+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:52:39.132903+0000', 'last_peered': '2024-09-16T02:52:39.132903+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:52:39.132903+0000', 'last_undegraded': '2024-09-16T02:52:39.132903+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7212, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 350, 'num_bytes_recovered': 350, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 350}, {'shards': '2,3', 'objects': 3606}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:42.625 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:42.625 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:42.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:43.031 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:43.031 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1422,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":384528384,"bytes_avail":192888999936,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":11236,"misplaced_total":63846,"misplaced_ratio":0.17598596623124393,"recovering_objects_per_sec":109,"recovering_bytes_per_sec":6101,"recovering_keys_per_sec":0,"num_objects_recovered":961,"num_bytes_recovered":53361,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:43.052 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:43.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:43.384 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:43.385 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:43.403 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2024-09-16T02:52:41.751839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11236,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23423,"num_bytes_recovered":193781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":375516,"kb_used_data":195132,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188368164,"statfs":{"total":193273528320,"available":192888999936,"internally_reserved":0,"allocated":199815168,"data_stored":613123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":53361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.745911"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":5093,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:39.132903+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:39.132903+0000","last_peered":"2024-09-16T02:52:39.132903+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:39.132903+0000","last_undegraded":"2024-09-16T02:52:39.132903+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7212,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":350},{"shards":"2,3","objects":3606}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4048,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017513+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017513+0000","last_peered":"2024-09-16T02:52:34.017513+0000","last_clean":"2024-09-16T02:52:34.017513+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017513+0000","last_undegraded":"2024-09-16T02:52:34.017513+0000","last_fullsized":"2024-09-16T02:52:34.017513+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":11236,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23423,"num_bytes_recovered":193781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362987520,"data_stored":756690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":196432,"kb_used_data":97216,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94175408,"statfs":{"total":96636764160,"available":96435617792,"internally_reserved":0,"allocated":99549184,"data_stored":306474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179084,"kb_used_data":97916,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94192756,"statfs":{"total":96636764160,"available":96453382144,"internally_reserved":0,"allocated":100265984,"data_stored":306649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100016128,"data_stored":194720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99229696,"data_stored":194528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:46.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:46.738 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:46.758 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2024-09-16T02:52:45.753054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10760,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23661,"num_bytes_recovered":194019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":376532,"kb_used_data":196148,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188367148,"statfs":{"total":193273528320,"available":192887959552,"internally_reserved":0,"allocated":200855552,"data_stored":613377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":1558,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":53599,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.745908"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":5807,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:44.153403+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:44.153403+0000","last_peered":"2024-09-16T02:52:44.153403+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:44.153403+0000","last_undegraded":"2024-09-16T02:52:44.153403+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6736,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":588},{"shards":"2,3","objects":3368}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4048,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017513+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017513+0000","last_peered":"2024-09-16T02:52:34.017513+0000","last_clean":"2024-09-16T02:52:34.017513+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017513+0000","last_undegraded":"2024-09-16T02:52:34.017513+0000","last_fullsized":"2024-09-16T02:52:34.017513+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10760,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23661,"num_bytes_recovered":194019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364019712,"data_stored":756942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640546,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":197448,"kb_used_data":98232,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94174392,"statfs":{"total":96636764160,"available":96434577408,"internally_reserved":0,"allocated":100589568,"data_stored":306728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179084,"kb_used_data":97916,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94192756,"statfs":{"total":96636764160,"available":96453382144,"internally_reserved":0,"allocated":100265984,"data_stored":306649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100016128,"data_stored":194720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":100261888,"data_stored":194780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:46.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:47.093 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:47.094 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:47.115 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2024-09-16T02:52:45.753054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10760,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23661,"num_bytes_recovered":194019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":376532,"kb_used_data":196148,"kb_used_omap":3,"kb_used_meta":180348,"kb_avail":188367148,"statfs":{"total":193273528320,"available":192887959552,"internally_reserved":0,"allocated":200855552,"data_stored":613377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":184677258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3956,"num_objects_misplaced":1558,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":53599,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.745908"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":5807,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:44.153403+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:44.153403+0000","last_peered":"2024-09-16T02:52:44.153403+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:44.153403+0000","last_undegraded":"2024-09-16T02:52:44.153403+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6736,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":588},{"shards":"2,3","objects":3368}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4048,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017513+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017513+0000","last_peered":"2024-09-16T02:52:34.017513+0000","last_clean":"2024-09-16T02:52:34.017513+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017513+0000","last_undegraded":"2024-09-16T02:52:34.017513+0000","last_fullsized":"2024-09-16T02:52:34.017513+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10760,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23661,"num_bytes_recovered":194019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":364019712,"data_stored":756942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640546,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":197448,"kb_used_data":98232,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94174392,"statfs":{"total":96636764160,"available":96434577408,"internally_reserved":0,"allocated":100589568,"data_stored":306728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179084,"kb_used_data":97916,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94192756,"statfs":{"total":96636764160,"available":96453382144,"internally_reserved":0,"allocated":100265984,"data_stored":306649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100016128,"data_stored":194720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":100261888,"data_stored":194780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:47.116 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:47.116 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 5807, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:44.153403+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:52:44.153403+0000', 'last_peered': '2024-09-16T02:52:44.153403+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:52:44.153403+0000', 'last_undegraded': '2024-09-16T02:52:44.153403+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6736, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 588, 'num_bytes_recovered': 588, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 588}, {'shards': '2,3', 'objects': 3368}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:47.116 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:47.116 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:47.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:47.527 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:47.527 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1427,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":385568768,"bytes_avail":192887959552,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":10760,"misplaced_total":63846,"misplaced_ratio":0.16853052657958212,"recovering_objects_per_sec":102,"recovering_bytes_per_sec":4563,"recovering_keys_per_sec":0,"num_objects_recovered":1199,"num_bytes_recovered":53599,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:47.549 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:47.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:47.884 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:47.884 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:47.904 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2024-09-16T02:52:47.753597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10760,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23661,"num_bytes_recovered":194019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":418784,"kb_used_data":198144,"kb_used_omap":3,"kb_used_meta":220604,"kb_avail":188324896,"statfs":{"total":193273528320,"available":192844693504,"internally_reserved":0,"allocated":202899456,"data_stored":613876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":225899402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2224,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":53512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003268"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":5807,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:44.153403+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:44.153403+0000","last_peered":"2024-09-16T02:52:44.153403+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:44.153403+0000","last_undegraded":"2024-09-16T02:52:44.153403+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6736,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":588},{"shards":"2,3","objects":3368}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10760,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23661,"num_bytes_recovered":194019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":366080000,"data_stored":757445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640547,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220692,"kb_used_data":99268,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94151148,"statfs":{"total":96636764160,"available":96410775552,"internally_reserved":0,"allocated":101650432,"data_stored":306987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673251,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":198092,"kb_used_data":98876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94173748,"statfs":{"total":96636764160,"available":96433917952,"internally_reserved":0,"allocated":101249024,"data_stored":306889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101019648,"data_stored":194965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101318656,"data_stored":195038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:50.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:51.236 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:51.236 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:51.255 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2024-09-16T02:52:49.753963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10260,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23911,"num_bytes_recovered":194269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":418784,"kb_used_data":198144,"kb_used_omap":3,"kb_used_meta":220604,"kb_avail":188324896,"statfs":{"total":193273528320,"available":192844693504,"internally_reserved":0,"allocated":202899456,"data_stored":613876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":225899402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1502,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":6557,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:49.174597+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:49.174597+0000","last_peered":"2024-09-16T02:52:49.174597+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:49.174597+0000","last_undegraded":"2024-09-16T02:52:49.174597+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6236,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":838},{"shards":"2,3","objects":3118}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10260,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23911,"num_bytes_recovered":194269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":366080000,"data_stored":757445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640547,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220692,"kb_used_data":99268,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94151148,"statfs":{"total":96636764160,"available":96410775552,"internally_reserved":0,"allocated":101650432,"data_stored":306987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673251,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":198092,"kb_used_data":98876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94173748,"statfs":{"total":96636764160,"available":96433917952,"internally_reserved":0,"allocated":101249024,"data_stored":306889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101019648,"data_stored":194965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101318656,"data_stored":195038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:51.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:51.606 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:51.625 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2024-09-16T02:52:49.753963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10260,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23911,"num_bytes_recovered":194269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":418784,"kb_used_data":198144,"kb_used_omap":3,"kb_used_meta":220604,"kb_avail":188324896,"statfs":{"total":193273528320,"available":192844693504,"internally_reserved":0,"allocated":202899456,"data_stored":613876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":225899402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1502,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":6557,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:49.174597+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:49.174597+0000","last_peered":"2024-09-16T02:52:49.174597+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:49.174597+0000","last_undegraded":"2024-09-16T02:52:49.174597+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6236,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":838},{"shards":"2,3","objects":3118}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10260,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23911,"num_bytes_recovered":194269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":366080000,"data_stored":757445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640547,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220692,"kb_used_data":99268,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94151148,"statfs":{"total":96636764160,"available":96410775552,"internally_reserved":0,"allocated":101650432,"data_stored":306987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673251,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":198092,"kb_used_data":98876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94173748,"statfs":{"total":96636764160,"available":96433917952,"internally_reserved":0,"allocated":101249024,"data_stored":306889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101019648,"data_stored":194965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101318656,"data_stored":195038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:51.626 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:51.626 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 6557, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:49.174597+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:52:49.174597+0000', 'last_peered': '2024-09-16T02:52:49.174597+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:52:49.174597+0000', 'last_undegraded': '2024-09-16T02:52:49.174597+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6236, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 838, 'num_bytes_recovered': 838, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 838}, {'shards': '2,3', 'objects': 3118}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:51.626 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:51.626 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:51.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:52.040 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:52.040 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1431,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":429858816,"bytes_avail":192843669504,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":10260,"misplaced_total":63846,"misplaced_ratio":0.16069918240766845,"recovering_objects_per_sec":40,"recovering_bytes_per_sec":40,"recovering_keys_per_sec":0,"num_objects_recovered":488,"num_bytes_recovered":488,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:52.060 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:52.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:52.377 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:52.378 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:52.395 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2024-09-16T02:52:51.754691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10260,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23911,"num_bytes_recovered":194269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":419784,"kb_used_data":199144,"kb_used_omap":3,"kb_used_meta":220604,"kb_avail":188323896,"statfs":{"total":193273528320,"available":192843669504,"internally_reserved":0,"allocated":203923456,"data_stored":614126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":225899402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-976,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003425"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":6557,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:49.174597+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:49.174597+0000","last_peered":"2024-09-16T02:52:49.174597+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:49.174597+0000","last_undegraded":"2024-09-16T02:52:49.174597+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6236,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":838,"num_bytes_recovered":838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":838},{"shards":"2,3","objects":3118}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":10260,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23911,"num_bytes_recovered":194269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":367210496,"data_stored":757721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640547,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":220692,"kb_used_data":99268,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94151148,"statfs":{"total":96636764160,"available":96410775552,"internally_reserved":0,"allocated":101650432,"data_stored":306987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199092,"kb_used_data":99876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94172748,"statfs":{"total":96636764160,"available":96432893952,"internally_reserved":0,"allocated":102273024,"data_stored":307139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986240,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102150144,"data_stored":195241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101318656,"data_stored":195038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:55.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:55.795 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:55.795 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:55.815 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-09-16T02:52:55.755903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9712,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24185,"num_bytes_recovered":194543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":420928,"kb_used_data":200288,"kb_used_omap":3,"kb_used_meta":220604,"kb_avail":188322752,"statfs":{"total":193273528320,"available":192842498048,"internally_reserved":0,"allocated":205094912,"data_stored":614412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":225899402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1524,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003428"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":7379,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:54.192667+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:54.192667+0000","last_peered":"2024-09-16T02:52:54.192667+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:54.192667+0000","last_undegraded":"2024-09-16T02:52:54.192667+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5688,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":1112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1112},{"shards":"2,3","objects":2844}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9712,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24185,"num_bytes_recovered":194543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368386048,"data_stored":758008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640548,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221836,"kb_used_data":100412,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94150004,"statfs":{"total":96636764160,"available":96409604096,"internally_reserved":0,"allocated":102821888,"data_stored":307273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199092,"kb_used_data":99876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94172748,"statfs":{"total":96636764160,"available":96432893952,"internally_reserved":0,"allocated":102273024,"data_stored":307139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986240,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102150144,"data_stored":195241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102494208,"data_stored":195325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:55.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:56.137 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:56.137 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:56.158 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-09-16T02:52:55.755903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9712,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24185,"num_bytes_recovered":194543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":420928,"kb_used_data":200288,"kb_used_omap":3,"kb_used_meta":220604,"kb_avail":188322752,"statfs":{"total":193273528320,"available":192842498048,"internally_reserved":0,"allocated":205094912,"data_stored":614412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":225899402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1524,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003428"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":7379,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:54.192667+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:54.192667+0000","last_peered":"2024-09-16T02:52:54.192667+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:54.192667+0000","last_undegraded":"2024-09-16T02:52:54.192667+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5688,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":1112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1112},{"shards":"2,3","objects":2844}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9712,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24185,"num_bytes_recovered":194543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368386048,"data_stored":758008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640548,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221836,"kb_used_data":100412,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94150004,"statfs":{"total":96636764160,"available":96409604096,"internally_reserved":0,"allocated":102821888,"data_stored":307273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199092,"kb_used_data":99876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94172748,"statfs":{"total":96636764160,"available":96432893952,"internally_reserved":0,"allocated":102273024,"data_stored":307139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986240,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102150144,"data_stored":195241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102494208,"data_stored":195325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:56.158 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:52:56.158 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 7379, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:54.192667+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:52:54.192667+0000', 'last_peered': '2024-09-16T02:52:54.192667+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:52:54.192667+0000', 'last_undegraded': '2024-09-16T02:52:54.192667+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5688, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1112, 'num_bytes_recovered': 1112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1112}, {'shards': '2,3', 'objects': 2844}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:56.159 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:52:56.159 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:52:56.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:52:56.614 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:56.614 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1436,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":431030272,"bytes_avail":192842498048,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":9712,"misplaced_total":63846,"misplaced_ratio":0.15211602919525108,"recovering_objects_per_sec":63,"recovering_bytes_per_sec":63,"recovering_keys_per_sec":0,"num_objects_recovered":762,"num_bytes_recovered":762,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:52:56.635 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:52:56.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:52:56.951 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:52:56.952 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:52:56.969 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-09-16T02:52:55.755903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9712,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24185,"num_bytes_recovered":194543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":420928,"kb_used_data":200288,"kb_used_omap":3,"kb_used_meta":220604,"kb_avail":188322752,"statfs":{"total":193273528320,"available":192842498048,"internally_reserved":0,"allocated":205094912,"data_stored":614412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":225899402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1524,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003428"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":7379,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:54.192667+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:54.192667+0000","last_peered":"2024-09-16T02:52:54.192667+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:54.192667+0000","last_undegraded":"2024-09-16T02:52:54.192667+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5688,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":1112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1112},{"shards":"2,3","objects":2844}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9712,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24185,"num_bytes_recovered":194543,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368386048,"data_stored":758008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640548,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221836,"kb_used_data":100412,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94150004,"statfs":{"total":96636764160,"available":96409604096,"internally_reserved":0,"allocated":102821888,"data_stored":307273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199092,"kb_used_data":99876,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":94172748,"statfs":{"total":96636764160,"available":96432893952,"internally_reserved":0,"allocated":102273024,"data_stored":307139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986240,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102150144,"data_stored":195241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102494208,"data_stored":195325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:52:59.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:00.337 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:00.338 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:00.358 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-09-16T02:52:59.757022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":427376,"kb_used_data":202576,"kb_used_omap":3,"kb_used_meta":224764,"kb_avail":188316304,"statfs":{"total":193273528320,"available":192835895296,"internally_reserved":0,"allocated":207437824,"data_stored":614984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":230159242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1624,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":812,"num_bytes_recovered":812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003425"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":8243,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:59.204789+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:59.204789+0000","last_peered":"2024-09-16T02:52:59.204789+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:59.204789+0000","last_undegraded":"2024-09-16T02:52:59.204789+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":1400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1400},{"shards":"2,3","objects":2556}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":370745344,"data_stored":758584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640549,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":222988,"kb_used_data":101564,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94148852,"statfs":{"total":96636764160,"available":96408424448,"internally_reserved":0,"allocated":104001536,"data_stored":307561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204388,"kb_used_data":101012,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94167452,"statfs":{"total":96636764160,"available":96427470848,"internally_reserved":0,"allocated":103436288,"data_stored":307423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986241,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103350272,"data_stored":195534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103653376,"data_stored":195608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:00.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:00.694 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:00.714 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-09-16T02:52:59.757022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":427376,"kb_used_data":202576,"kb_used_omap":3,"kb_used_meta":224764,"kb_avail":188316304,"statfs":{"total":193273528320,"available":192835895296,"internally_reserved":0,"allocated":207437824,"data_stored":614984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":230159242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1624,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":812,"num_bytes_recovered":812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003425"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":8243,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:59.204789+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:59.204789+0000","last_peered":"2024-09-16T02:52:59.204789+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:59.204789+0000","last_undegraded":"2024-09-16T02:52:59.204789+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":1400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1400},{"shards":"2,3","objects":2556}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":370745344,"data_stored":758584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640549,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":222988,"kb_used_data":101564,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94148852,"statfs":{"total":96636764160,"available":96408424448,"internally_reserved":0,"allocated":104001536,"data_stored":307561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204388,"kb_used_data":101012,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94167452,"statfs":{"total":96636764160,"available":96427470848,"internally_reserved":0,"allocated":103436288,"data_stored":307423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986241,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103350272,"data_stored":195534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103653376,"data_stored":195608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:00.715 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:53:00.715 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 8243, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:59.204789+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:52:59.204789+0000', 'last_peered': '2024-09-16T02:52:59.204789+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:52:59.204789+0000', 'last_undegraded': '2024-09-16T02:52:59.204789+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5112, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1400, 'num_bytes_recovered': 1400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1400}, {'shards': '2,3', 'objects': 2556}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:00.715 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:00.715 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:00.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:01.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:01.127 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1440,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":437633024,"bytes_avail":192835895296,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":9136,"misplaced_total":63846,"misplaced_ratio":0.14309432070920652,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":67,"recovering_keys_per_sec":0,"num_objects_recovered":812,"num_bytes_recovered":812,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:01.148 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:01.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:01.485 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:01.485 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:01.505 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-09-16T02:52:59.757022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":427376,"kb_used_data":202576,"kb_used_omap":3,"kb_used_meta":224764,"kb_avail":188316304,"statfs":{"total":193273528320,"available":192835895296,"internally_reserved":0,"allocated":207437824,"data_stored":614984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":230159242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1624,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":812,"num_bytes_recovered":812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003425"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":8243,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:59.204789+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:59.204789+0000","last_peered":"2024-09-16T02:52:59.204789+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:59.204789+0000","last_undegraded":"2024-09-16T02:52:59.204789+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":1400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1400},{"shards":"2,3","objects":2556}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":370745344,"data_stored":758584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640549,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":222988,"kb_used_data":101564,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94148852,"statfs":{"total":96636764160,"available":96408424448,"internally_reserved":0,"allocated":104001536,"data_stored":307561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673253,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204388,"kb_used_data":101012,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94167452,"statfs":{"total":96636764160,"available":96427470848,"internally_reserved":0,"allocated":103436288,"data_stored":307423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986241,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103350272,"data_stored":195534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103653376,"data_stored":195608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:04.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:04.841 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:04.860 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":757,"stamp":"2024-09-16T02:53:03.758390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":429676,"kb_used_data":204876,"kb_used_omap":3,"kb_used_meta":224764,"kb_avail":188314004,"statfs":{"total":193273528320,"available":192833540096,"internally_reserved":0,"allocated":209793024,"data_stored":615559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":230159242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1124,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":562,"num_bytes_recovered":562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003699"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":8243,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:59.204789+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:59.204789+0000","last_peered":"2024-09-16T02:52:59.204789+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:59.204789+0000","last_undegraded":"2024-09-16T02:52:59.204789+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":1400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1400},{"shards":"2,3","objects":2556}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":373338112,"data_stored":759217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640550,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":224024,"kb_used_data":102600,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94147816,"statfs":{"total":96636764160,"available":96407363584,"internally_reserved":0,"allocated":105062400,"data_stored":307820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673254,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":205652,"kb_used_data":102276,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94166188,"statfs":{"total":96636764160,"available":96426176512,"internally_reserved":0,"allocated":104730624,"data_stored":307739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986242,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":104689664,"data_stored":195861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104906752,"data_stored":195914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:04.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:05.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:05.190 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:05.210 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":757,"stamp":"2024-09-16T02:53:03.758390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":429676,"kb_used_data":204876,"kb_used_omap":3,"kb_used_meta":224764,"kb_avail":188314004,"statfs":{"total":193273528320,"available":192833540096,"internally_reserved":0,"allocated":209793024,"data_stored":615559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":230159242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1124,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":562,"num_bytes_recovered":562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003699"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":8243,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:52:59.204789+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:52:59.204789+0000","last_peered":"2024-09-16T02:52:59.204789+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:52:59.204789+0000","last_undegraded":"2024-09-16T02:52:59.204789+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":1400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1400},{"shards":"2,3","objects":2556}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":9136,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24473,"num_bytes_recovered":194831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":373338112,"data_stored":759217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640550,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":224024,"kb_used_data":102600,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94147816,"statfs":{"total":96636764160,"available":96407363584,"internally_reserved":0,"allocated":105062400,"data_stored":307820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673254,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":205652,"kb_used_data":102276,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94166188,"statfs":{"total":96636764160,"available":96426176512,"internally_reserved":0,"allocated":104730624,"data_stored":307739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986242,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":104689664,"data_stored":195861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104906752,"data_stored":195914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:05.211 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:53:05.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 8243, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:52:59.204789+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:52:59.204789+0000', 'last_peered': '2024-09-16T02:52:59.204789+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:52:59.204789+0000', 'last_undegraded': '2024-09-16T02:52:59.204789+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5112, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1400, 'num_bytes_recovered': 1400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1400}, {'shards': '2,3', 'objects': 2556}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:05.211 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:05.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:05.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:05.634 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:05.634 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1445,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":439988224,"bytes_avail":192833540096,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":9136,"misplaced_total":63846,"misplaced_ratio":0.14309432070920652,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":46,"recovering_keys_per_sec":0,"num_objects_recovered":562,"num_bytes_recovered":562,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:05.657 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:05.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:06.024 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:06.024 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:06.044 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":758,"stamp":"2024-09-16T02:53:05.758990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":8516,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24783,"num_bytes_recovered":195141,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":429676,"kb_used_data":204876,"kb_used_omap":3,"kb_used_meta":224764,"kb_avail":188314004,"statfs":{"total":193273528320,"available":192833540096,"internally_reserved":0,"allocated":209793024,"data_stored":615559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":230159242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1744,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003849"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":9173,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:04.202424+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:04.202424+0000","last_peered":"2024-09-16T02:53:04.202424+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:04.202424+0000","last_undegraded":"2024-09-16T02:53:04.202424+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4492,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1710,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1710},{"shards":"2,3","objects":2246}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":8516,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24783,"num_bytes_recovered":195141,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":373338112,"data_stored":759217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640550,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":224024,"kb_used_data":102600,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94147816,"statfs":{"total":96636764160,"available":96407363584,"internally_reserved":0,"allocated":105062400,"data_stored":307820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673254,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":205652,"kb_used_data":102276,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94166188,"statfs":{"total":96636764160,"available":96426176512,"internally_reserved":0,"allocated":104730624,"data_stored":307739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986242,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":104689664,"data_stored":195861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104906752,"data_stored":195914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:09.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:09.390 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:09.410 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2024-09-16T02:53:07.759447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":8516,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24783,"num_bytes_recovered":195141,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":436668,"kb_used_data":207708,"kb_used_omap":3,"kb_used_meta":228924,"kb_avail":188307012,"statfs":{"total":193273528320,"available":192826380288,"internally_reserved":0,"allocated":212692992,"data_stored":616267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":234419082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1196,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":598,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":9173,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:04.202424+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:04.202424+0000","last_peered":"2024-09-16T02:53:04.202424+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:04.202424+0000","last_undegraded":"2024-09-16T02:53:04.202424+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4492,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1710,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1710},{"shards":"2,3","objects":2246}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":8516,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24783,"num_bytes_recovered":195141,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376020992,"data_stored":759872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640551,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229680,"kb_used_data":104096,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94142160,"statfs":{"total":96636764160,"available":96401571840,"internally_reserved":0,"allocated":106594304,"data_stored":308194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673255,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":206988,"kb_used_data":103612,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94164852,"statfs":{"total":96636764160,"available":96424808448,"internally_reserved":0,"allocated":106098688,"data_stored":308073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986243,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106033152,"data_stored":196189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106246144,"data_stored":196241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:09.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:09.787 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:09.787 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:09.809 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":760,"stamp":"2024-09-16T02:53:09.760025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":436668,"kb_used_data":207708,"kb_used_omap":3,"kb_used_meta":228924,"kb_avail":188307012,"statfs":{"total":193273528320,"available":192826380288,"internally_reserved":0,"allocated":212692992,"data_stored":616267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":234419082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1880,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003579"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":10199,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:09.218138+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:09.218138+0000","last_peered":"2024-09-16T02:53:09.218138+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:09.218138+0000","last_undegraded":"2024-09-16T02:53:09.218138+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3808,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1904}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376020992,"data_stored":759872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640551,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229680,"kb_used_data":104096,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94142160,"statfs":{"total":96636764160,"available":96401571840,"internally_reserved":0,"allocated":106594304,"data_stored":308194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673255,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":206988,"kb_used_data":103612,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94164852,"statfs":{"total":96636764160,"available":96424808448,"internally_reserved":0,"allocated":106098688,"data_stored":308073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986243,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106033152,"data_stored":196189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106246144,"data_stored":196241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:09.810 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:53:09.810 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 10199, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:09.218138+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:53:09.218138+0000', 'last_peered': '2024-09-16T02:53:09.218138+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:53:09.218138+0000', 'last_undegraded': '2024-09-16T02:53:09.218138+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3808, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2052, 'num_bytes_recovered': 2052, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2052}, {'shards': '2,3', 'objects': 1904}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:09.810 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:09.810 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:09.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:10.218 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:10.218 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1449,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":447148032,"bytes_avail":192826380288,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":7832,"misplaced_total":63846,"misplaced_ratio":0.12267017510885568,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":78,"recovering_keys_per_sec":0,"num_objects_recovered":940,"num_bytes_recovered":940,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:10.239 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:10.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:10.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:10.581 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:10.601 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":760,"stamp":"2024-09-16T02:53:09.760025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":436668,"kb_used_data":207708,"kb_used_omap":3,"kb_used_meta":228924,"kb_avail":188307012,"statfs":{"total":193273528320,"available":192826380288,"internally_reserved":0,"allocated":212692992,"data_stored":616267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":234419082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1880,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003579"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":10199,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:09.218138+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:09.218138+0000","last_peered":"2024-09-16T02:53:09.218138+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:09.218138+0000","last_undegraded":"2024-09-16T02:53:09.218138+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3808,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1904}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376020992,"data_stored":759872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640551,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229680,"kb_used_data":104096,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94142160,"statfs":{"total":96636764160,"available":96401571840,"internally_reserved":0,"allocated":106594304,"data_stored":308194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673255,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":206988,"kb_used_data":103612,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":94164852,"statfs":{"total":96636764160,"available":96424808448,"internally_reserved":0,"allocated":106098688,"data_stored":308073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986243,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106033152,"data_stored":196189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106246144,"data_stored":196241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:13.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:13.971 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:13.971 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:13.990 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":762,"stamp":"2024-09-16T02:53:13.761319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":443792,"kb_used_data":210672,"kb_used_omap":3,"kb_used_meta":233084,"kb_avail":188299888,"statfs":{"total":193273528320,"available":192819085312,"internally_reserved":0,"allocated":215728128,"data_stored":617008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":238678922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1304,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":652,"num_bytes_recovered":652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003673"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":10199,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:09.218138+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:09.218138+0000","last_peered":"2024-09-16T02:53:09.218138+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:09.218138+0000","last_undegraded":"2024-09-16T02:53:09.218138+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3808,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1904}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":379002880,"data_stored":760600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640552,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231160,"kb_used_data":105576,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94140680,"statfs":{"total":96636764160,"available":96400056320,"internally_reserved":0,"allocated":108109824,"data_stored":308564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673256,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212632,"kb_used_data":105096,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94159208,"statfs":{"total":96636764160,"available":96419028992,"internally_reserved":0,"allocated":107618304,"data_stored":308444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986244,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107507712,"data_stored":196549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107753472,"data_stored":196609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:13.991 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:14.323 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:14.323 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:14.343 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":762,"stamp":"2024-09-16T02:53:13.761319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":443792,"kb_used_data":210672,"kb_used_omap":3,"kb_used_meta":233084,"kb_avail":188299888,"statfs":{"total":193273528320,"available":192819085312,"internally_reserved":0,"allocated":215728128,"data_stored":617008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":238678922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1304,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":652,"num_bytes_recovered":652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003673"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":10199,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:09.218138+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:09.218138+0000","last_peered":"2024-09-16T02:53:09.218138+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:09.218138+0000","last_undegraded":"2024-09-16T02:53:09.218138+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3808,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1904}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":379002880,"data_stored":760600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640552,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231160,"kb_used_data":105576,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94140680,"statfs":{"total":96636764160,"available":96400056320,"internally_reserved":0,"allocated":108109824,"data_stored":308564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673256,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212632,"kb_used_data":105096,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94159208,"statfs":{"total":96636764160,"available":96419028992,"internally_reserved":0,"allocated":107618304,"data_stored":308444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986244,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107507712,"data_stored":196549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107753472,"data_stored":196609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:14.344 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:53:14.345 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 10199, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:09.218138+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:53:09.218138+0000', 'last_peered': '2024-09-16T02:53:09.218138+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:53:09.218138+0000', 'last_undegraded': '2024-09-16T02:53:09.218138+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3808, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2052, 'num_bytes_recovered': 2052, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2052}, {'shards': '2,3', 'objects': 1904}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:14.345 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:14.345 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:14.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:14.765 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:14.765 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1454,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":454443008,"bytes_avail":192819085312,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":7832,"misplaced_total":63846,"misplaced_ratio":0.12267017510885568,"recovering_objects_per_sec":54,"recovering_bytes_per_sec":54,"recovering_keys_per_sec":0,"num_objects_recovered":652,"num_bytes_recovered":652,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:14.789 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:14.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:15.138 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:15.139 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:15.159 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":762,"stamp":"2024-09-16T02:53:13.761319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":443792,"kb_used_data":210672,"kb_used_omap":3,"kb_used_meta":233084,"kb_avail":188299888,"statfs":{"total":193273528320,"available":192819085312,"internally_reserved":0,"allocated":215728128,"data_stored":617008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":238678922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1304,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":652,"num_bytes_recovered":652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003673"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":10199,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:09.218138+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:09.218138+0000","last_peered":"2024-09-16T02:53:09.218138+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:09.218138+0000","last_undegraded":"2024-09-16T02:53:09.218138+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3808,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1904}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7832,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25125,"num_bytes_recovered":195483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":379002880,"data_stored":760600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640552,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231160,"kb_used_data":105576,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94140680,"statfs":{"total":96636764160,"available":96400056320,"internally_reserved":0,"allocated":108109824,"data_stored":308564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673256,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212632,"kb_used_data":105096,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94159208,"statfs":{"total":96636764160,"available":96419028992,"internally_reserved":0,"allocated":107618304,"data_stored":308444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986244,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107507712,"data_stored":196549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107753472,"data_stored":196609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:18.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:18.511 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:18.512 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:18.532 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-09-16T02:53:17.762599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7014,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25534,"num_bytes_recovered":195892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":451060,"kb_used_data":213780,"kb_used_omap":3,"kb_used_meta":237244,"kb_avail":188292620,"statfs":{"total":193273528320,"available":192811642880,"internally_reserved":0,"allocated":218910720,"data_stored":617785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":242938762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1502,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003609"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":11426,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:14.221683+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:14.221683+0000","last_peered":"2024-09-16T02:53:14.221683+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:14.221683+0000","last_undegraded":"2024-09-16T02:53:14.221683+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2990,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2461,"num_bytes_recovered":2461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2461},{"shards":"2,3","objects":1495}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7014,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25534,"num_bytes_recovered":195892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":382464000,"data_stored":761445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236772,"kb_used_data":107028,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94135068,"statfs":{"total":96636764160,"available":96394309632,"internally_reserved":0,"allocated":109596672,"data_stored":308927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673257,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":214288,"kb_used_data":106752,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94157552,"statfs":{"total":96636764160,"available":96417333248,"internally_reserved":0,"allocated":109314048,"data_stored":308858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986245,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109219840,"data_stored":196967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109502464,"data_stored":197036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:18.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:18.857 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:18.857 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:18.877 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-09-16T02:53:17.762599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7014,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25534,"num_bytes_recovered":195892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":451060,"kb_used_data":213780,"kb_used_omap":3,"kb_used_meta":237244,"kb_avail":188292620,"statfs":{"total":193273528320,"available":192811642880,"internally_reserved":0,"allocated":218910720,"data_stored":617785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":242938762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1502,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003609"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":11426,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:14.221683+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:14.221683+0000","last_peered":"2024-09-16T02:53:14.221683+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:14.221683+0000","last_undegraded":"2024-09-16T02:53:14.221683+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2990,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2461,"num_bytes_recovered":2461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2461},{"shards":"2,3","objects":1495}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7014,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25534,"num_bytes_recovered":195892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":382464000,"data_stored":761445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236772,"kb_used_data":107028,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94135068,"statfs":{"total":96636764160,"available":96394309632,"internally_reserved":0,"allocated":109596672,"data_stored":308927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673257,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":214288,"kb_used_data":106752,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94157552,"statfs":{"total":96636764160,"available":96417333248,"internally_reserved":0,"allocated":109314048,"data_stored":308858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986245,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109219840,"data_stored":196967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109502464,"data_stored":197036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:18.878 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:53:18.878 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 11426, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:14.221683+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:53:14.221683+0000', 'last_peered': '2024-09-16T02:53:14.221683+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:53:14.221683+0000', 'last_undegraded': '2024-09-16T02:53:14.221683+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2990, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2461, 'num_bytes_recovered': 2461, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2461}, {'shards': '2,3', 'objects': 1495}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:18.878 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:18.878 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:18.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:19.286 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:19.287 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1458,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":461885440,"bytes_avail":192811642880,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":7014,"misplaced_total":63846,"misplaced_ratio":0.10985809604360491,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":62,"recovering_keys_per_sec":0,"num_objects_recovered":751,"num_bytes_recovered":751,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:19.305 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:19.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:19.637 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:19.638 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:19.658 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-09-16T02:53:17.762599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7014,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25534,"num_bytes_recovered":195892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":451060,"kb_used_data":213780,"kb_used_omap":3,"kb_used_meta":237244,"kb_avail":188292620,"statfs":{"total":193273528320,"available":192811642880,"internally_reserved":0,"allocated":218910720,"data_stored":617785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":242938762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1502,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":751,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003609"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":11426,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:14.221683+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:14.221683+0000","last_peered":"2024-09-16T02:53:14.221683+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:14.221683+0000","last_undegraded":"2024-09-16T02:53:14.221683+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2990,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2461,"num_bytes_recovered":2461,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2461},{"shards":"2,3","objects":1495}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":7014,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25534,"num_bytes_recovered":195892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":382464000,"data_stored":761445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236772,"kb_used_data":107028,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94135068,"statfs":{"total":96636764160,"available":96394309632,"internally_reserved":0,"allocated":109596672,"data_stored":308927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673257,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":214288,"kb_used_data":106752,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":94157552,"statfs":{"total":96636764160,"available":96417333248,"internally_reserved":0,"allocated":109314048,"data_stored":308858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986245,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109219840,"data_stored":196967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109502464,"data_stored":197036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:22.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:23.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:23.028 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:23.048 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-16T02:53:21.763754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":6018,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":196390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":475284,"kb_used_data":215796,"kb_used_omap":3,"kb_used_meta":259452,"kb_avail":188268396,"statfs":{"total":193273528320,"available":192786837504,"internally_reserved":0,"allocated":220975104,"data_stored":618289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":265679754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1814,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":907,"num_bytes_recovered":907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003729"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":12920,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:19.222115+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:19.222115+0000","last_peered":"2024-09-16T02:53:19.222115+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:19.222115+0000","last_undegraded":"2024-09-16T02:53:19.222115+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1994,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2959,"num_bytes_recovered":2959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2959},{"shards":"2,3","objects":997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":6018,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":196390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":384598016,"data_stored":761966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236772,"kb_used_data":107028,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94135068,"statfs":{"total":96636764160,"available":96394309632,"internally_reserved":0,"allocated":109596672,"data_stored":308927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673258,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238512,"kb_used_data":108768,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94133328,"statfs":{"total":96636764160,"available":96392527872,"internally_reserved":0,"allocated":111378432,"data_stored":309362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986246,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":111353856,"data_stored":197488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109502464,"data_stored":197036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:23.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:23.411 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-16T02:53:21.763754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":6018,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":196390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":475284,"kb_used_data":215796,"kb_used_omap":3,"kb_used_meta":259452,"kb_avail":188268396,"statfs":{"total":193273528320,"available":192786837504,"internally_reserved":0,"allocated":220975104,"data_stored":618289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":265679754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1814,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":907,"num_bytes_recovered":907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003729"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":12920,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:19.222115+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:19.222115+0000","last_peered":"2024-09-16T02:53:19.222115+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:19.222115+0000","last_undegraded":"2024-09-16T02:53:19.222115+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1994,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2959,"num_bytes_recovered":2959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2959},{"shards":"2,3","objects":997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":6018,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":196390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":384598016,"data_stored":761966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":236772,"kb_used_data":107028,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94135068,"statfs":{"total":96636764160,"available":96394309632,"internally_reserved":0,"allocated":109596672,"data_stored":308927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673258,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238512,"kb_used_data":108768,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94133328,"statfs":{"total":96636764160,"available":96392527872,"internally_reserved":0,"allocated":111378432,"data_stored":309362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986246,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":111353856,"data_stored":197488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":109502464,"data_stored":197036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:23.411 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:53:23.411 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 12920, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:19.222115+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:53:19.222115+0000', 'last_peered': '2024-09-16T02:53:19.222115+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:53:19.222115+0000', 'last_undegraded': '2024-09-16T02:53:19.222115+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1994, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2959, 'num_bytes_recovered': 2959, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2959}, {'shards': '2,3', 'objects': 997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:23.412 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:23.412 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:23.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:23.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:23.848 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1463,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":507310080,"bytes_avail":192766218240,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":6018,"misplaced_total":63846,"misplaced_ratio":0.094258058453152896,"recovering_objects_per_sec":75,"recovering_bytes_per_sec":75,"recovering_keys_per_sec":0,"num_objects_recovered":907,"num_bytes_recovered":907,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:23.871 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:23.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:24.219 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:24.220 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:24.242 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":767,"stamp":"2024-09-16T02:53:23.764388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":6018,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":196390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":495420,"kb_used_data":217884,"kb_used_omap":3,"kb_used_meta":277500,"kb_avail":188248260,"statfs":{"total":193273528320,"available":192766218240,"internally_reserved":0,"allocated":223113216,"data_stored":618811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":284160906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1814,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":907,"num_bytes_recovered":907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003743"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":12920,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:19.222115+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:19.222115+0000","last_peered":"2024-09-16T02:53:19.222115+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:19.222115+0000","last_undegraded":"2024-09-16T02:53:19.222115+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1994,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2959,"num_bytes_recovered":2959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2959},{"shards":"2,3","objects":997}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3990,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017162+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017162+0000","last_peered":"2024-09-16T02:52:34.017162+0000","last_clean":"2024-09-16T02:52:34.017162+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017162+0000","last_undegraded":"2024-09-16T02:52:34.017162+0000","last_fullsized":"2024-09-16T02:52:34.017162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":6018,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26032,"num_bytes_recovered":196390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":386826240,"data_stored":762510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256908,"kb_used_data":109116,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94114932,"statfs":{"total":96636764160,"available":96373690368,"internally_reserved":0,"allocated":111734784,"data_stored":309449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673258,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238512,"kb_used_data":108768,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94133328,"statfs":{"total":96636764160,"available":96392527872,"internally_reserved":0,"allocated":111378432,"data_stored":309362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986246,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":111353856,"data_stored":197488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111730688,"data_stored":197580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:27.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:27.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:27.576 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:27.597 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":769,"stamp":"2024-09-16T02:53:26.832092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4650,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26716,"num_bytes_recovered":197074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":498440,"kb_used_data":220904,"kb_used_omap":3,"kb_used_meta":277500,"kb_avail":188245240,"statfs":{"total":193273528320,"available":192763125760,"internally_reserved":0,"allocated":226205696,"data_stored":619566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":284160906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2364,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":1182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.070082"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4003,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.011943+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:52:34.011943+0000","last_peered":"2024-09-16T02:52:34.011943+0000","last_clean":"2024-09-16T02:52:34.011943+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:52:34.011943+0000","last_undegraded":"2024-09-16T02:52:34.011943+0000","last_fullsized":"2024-09-16T02:52:34.011943+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":14972,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:24.227252+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:24.227252+0000","last_peered":"2024-09-16T02:53:24.227252+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:24.227252+0000","last_undegraded":"2024-09-16T02:53:24.227252+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":626,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3643,"num_bytes_recovered":3643,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3643},{"shards":"2,3","objects":313}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3991,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017219+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017219+0000","last_peered":"2024-09-16T02:52:34.017219+0000","last_clean":"2024-09-16T02:52:34.017219+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017219+0000","last_undegraded":"2024-09-16T02:52:34.017219+0000","last_fullsized":"2024-09-16T02:52:34.017219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4076,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.021618+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:52:34.021618+0000","last_peered":"2024-09-16T02:52:34.021618+0000","last_clean":"2024-09-16T02:52:34.021618+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:52:34.021618+0000","last_undegraded":"2024-09-16T02:52:34.021618+0000","last_fullsized":"2024-09-16T02:52:34.021618+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4650,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26716,"num_bytes_recovered":197074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":390012928,"data_stored":815676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256908,"kb_used_data":109116,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94114932,"statfs":{"total":96636764160,"available":96373690368,"internally_reserved":0,"allocated":111734784,"data_stored":309449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673259,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241532,"kb_used_data":111788,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94130308,"statfs":{"total":96636764160,"available":96389435392,"internally_reserved":0,"allocated":114470912,"data_stored":310117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114540544,"data_stored":250654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111730688,"data_stored":197580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:27.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:27.948 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:27.948 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:27.969 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":771,"stamp":"2024-09-16T02:53:27.850891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4650,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26716,"num_bytes_recovered":197074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":489068,"kb_used_data":224012,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188254612,"statfs":{"total":193273528320,"available":192772722688,"internally_reserved":0,"allocated":229388288,"data_stored":672731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1368,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.087588"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835726+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:26.835726+0000","last_peered":"2024-09-16T02:53:26.835726+0000","last_clean":"2024-09-16T02:53:26.835726+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:26.835726+0000","last_undegraded":"2024-09-16T02:53:26.835726+0000","last_fullsized":"2024-09-16T02:53:26.835726+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":14972,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:24.227252+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:24.227252+0000","last_peered":"2024-09-16T02:53:24.227252+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:24.227252+0000","last_undegraded":"2024-09-16T02:53:24.227252+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":626,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3643,"num_bytes_recovered":3643,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3643},{"shards":"2,3","objects":313}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3991,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017219+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017219+0000","last_peered":"2024-09-16T02:52:34.017219+0000","last_clean":"2024-09-16T02:52:34.017219+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017219+0000","last_undegraded":"2024-09-16T02:52:34.017219+0000","last_fullsized":"2024-09-16T02:52:34.017219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4078,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835101+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:26.835101+0000","last_peered":"2024-09-16T02:53:26.835101+0000","last_clean":"2024-09-16T02:53:26.835101+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:26.835101+0000","last_undegraded":"2024-09-16T02:53:26.835101+0000","last_fullsized":"2024-09-16T02:53:26.835101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4650,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26716,"num_bytes_recovered":197074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393011200,"data_stored":868796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247536,"kb_used_data":112224,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94124304,"statfs":{"total":96636764160,"available":96383287296,"internally_reserved":0,"allocated":114917376,"data_stored":362614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673259,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241532,"kb_used_data":111788,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94130308,"statfs":{"total":96636764160,"available":96389435392,"internally_reserved":0,"allocated":114470912,"data_stored":310117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114540544,"data_stored":250654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114728960,"data_stored":250700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:27.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-16T02:53:27.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "19'3956", 'reported_seq': 14972, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:24.227252+0000', 'last_change': '2024-09-16T02:52:32.965852+0000', 'last_active': '2024-09-16T02:53:24.227252+0000', 'last_peered': '2024-09-16T02:53:24.227252+0000', 'last_clean': '2024-09-16T02:46:37.354625+0000', 'last_became_active': '2024-09-16T02:47:26.523954+0000', 'last_became_peered': '2024-09-16T02:47:26.523954+0000', 'last_unstale': '2024-09-16T02:53:24.227252+0000', 'last_undegraded': '2024-09-16T02:53:24.227252+0000', 'last_fullsized': '2024-09-16T02:47:26.333736+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 3956, 'log_dups_size': 0, 'ondisk_log_size': 3956, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.001597977, 'stat_sum': {'num_bytes': 56356, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 7912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 626, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3956, 'num_write_kb': 4004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3643, 'num_bytes_recovered': 3643, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3643}, {'shards': '2,3', 'objects': 313}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:27.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:27.971 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4099, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T02:52:34.010633+0000', 'last_change': '2024-09-16T02:47:26.534038+0000', 'last_active': '2024-09-16T02:52:34.010633+0000', 'last_peered': '2024-09-16T02:52:34.010633+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:52:34.010633+0000', 'last_undegraded': '2024-09-16T02:47:26.520901+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4024, 'num_objects_misplaced': 4024, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:27.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:28.399 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4024/63846 objects degraded (6.303%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1467,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":500805632,"bytes_avail":192772722688,"bytes_total":193273528320,"degraded_objects":4024,"degraded_total":63846,"degraded_ratio":0.063026657895561194,"misplaced_objects":4650,"misplaced_total":63846,"misplaced_ratio":0.072831500798797105,"recovering_objects_per_sec":118,"recovering_bytes_per_sec":118,"recovering_keys_per_sec":0,"num_objects_recovered":1182,"num_bytes_recovered":1182,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:28.421 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:28.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:28.783 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:28.783 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:28.804 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":771,"stamp":"2024-09-16T02:53:27.850891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4650,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26716,"num_bytes_recovered":197074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":489068,"kb_used_data":224012,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188254612,"statfs":{"total":193273528320,"available":192772722688,"internally_reserved":0,"allocated":229388288,"data_stored":672731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1368,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":684,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.087588"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835726+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:26.835726+0000","last_peered":"2024-09-16T02:53:26.835726+0000","last_clean":"2024-09-16T02:53:26.835726+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:26.835726+0000","last_undegraded":"2024-09-16T02:53:26.835726+0000","last_fullsized":"2024-09-16T02:53:26.835726+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4060,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017496+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:52:34.017496+0000","last_peered":"2024-09-16T02:52:34.017496+0000","last_clean":"2024-09-16T02:52:34.017496+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:52:34.017496+0000","last_undegraded":"2024-09-16T02:52:34.017496+0000","last_fullsized":"2024-09-16T02:52:34.017496+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4119,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017282+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:52:34.017282+0000","last_peered":"2024-09-16T02:52:34.017282+0000","last_clean":"2024-09-16T02:52:34.017282+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:52:34.017282+0000","last_undegraded":"2024-09-16T02:52:34.017282+0000","last_fullsized":"2024-09-16T02:52:34.017282+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":14972,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:24.227252+0000","last_change":"2024-09-16T02:52:32.965852+0000","last_active":"2024-09-16T02:53:24.227252+0000","last_peered":"2024-09-16T02:53:24.227252+0000","last_clean":"2024-09-16T02:46:37.354625+0000","last_became_active":"2024-09-16T02:47:26.523954+0000","last_became_peered":"2024-09-16T02:47:26.523954+0000","last_unstale":"2024-09-16T02:53:24.227252+0000","last_undegraded":"2024-09-16T02:53:24.227252+0000","last_fullsized":"2024-09-16T02:47:26.333736+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":626,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3643,"num_bytes_recovered":3643,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3643},{"shards":"2,3","objects":313}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4099,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T02:52:34.010633+0000","last_change":"2024-09-16T02:47:26.534038+0000","last_active":"2024-09-16T02:52:34.010633+0000","last_peered":"2024-09-16T02:52:34.010633+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:52:34.010633+0000","last_undegraded":"2024-09-16T02:47:26.520901+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4024,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4049,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017564+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:52:34.017564+0000","last_peered":"2024-09-16T02:52:34.017564+0000","last_clean":"2024-09-16T02:52:34.017564+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:52:34.017564+0000","last_undegraded":"2024-09-16T02:52:34.017564+0000","last_fullsized":"2024-09-16T02:52:34.017564+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3991,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T02:52:34.017219+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:52:34.017219+0000","last_peered":"2024-09-16T02:52:34.017219+0000","last_clean":"2024-09-16T02:52:34.017219+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:52:34.017219+0000","last_undegraded":"2024-09-16T02:52:34.017219+0000","last_fullsized":"2024-09-16T02:52:34.017219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4078,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835101+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:26.835101+0000","last_peered":"2024-09-16T02:53:26.835101+0000","last_clean":"2024-09-16T02:53:26.835101+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:26.835101+0000","last_undegraded":"2024-09-16T02:53:26.835101+0000","last_fullsized":"2024-09-16T02:53:26.835101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4024,"num_objects_misplaced":4650,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26716,"num_bytes_recovered":197074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393011200,"data_stored":868796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247536,"kb_used_data":112224,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94124304,"statfs":{"total":96636764160,"available":96383287296,"internally_reserved":0,"allocated":114917376,"data_stored":362614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673259,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241532,"kb_used_data":111788,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94130308,"statfs":{"total":96636764160,"available":96389435392,"internally_reserved":0,"allocated":114470912,"data_stored":310117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114540544,"data_stored":250654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114728960,"data_stored":250700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32755712,"data_stored":86579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:31.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:32.138 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:32.158 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":773,"stamp":"2024-09-16T02:53:31.766619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27194,"num_bytes_recovered":249951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":494460,"kb_used_data":225244,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188249220,"statfs":{"total":193273528320,"available":192767201280,"internally_reserved":0,"allocated":230649856,"data_stored":730762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":1700,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":53561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002231"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835726+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:26.835726+0000","last_peered":"2024-09-16T02:53:26.835726+0000","last_clean":"2024-09-16T02:53:26.835726+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:26.835726+0000","last_undegraded":"2024-09-16T02:53:26.835726+0000","last_fullsized":"2024-09-16T02:53:26.835726+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4599,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:29.212372+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:29.212372+0000","last_peered":"2024-09-16T02:53:29.212372+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:29.212372+0000","last_undegraded":"2024-09-16T02:53:29.212372+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":165},{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4078,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835101+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:26.835101+0000","last_peered":"2024-09-16T02:53:26.835101+0000","last_clean":"2024-09-16T02:53:26.835101+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:26.835101+0000","last_undegraded":"2024-09-16T02:53:26.835101+0000","last_fullsized":"2024-09-16T02:53:26.835101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27194,"num_bytes_recovered":249951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377704448,"data_stored":812671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247536,"kb_used_data":112224,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94124304,"statfs":{"total":96636764160,"available":96383287296,"internally_reserved":0,"allocated":114917376,"data_stored":362614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673260,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246924,"kb_used_data":113020,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":94124916,"statfs":{"total":96636764160,"available":96383913984,"internally_reserved":0,"allocated":115732480,"data_stored":368148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115486720,"data_stored":250885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114728960,"data_stored":250700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:32.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:32.495 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:32.495 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:32.517 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":773,"stamp":"2024-09-16T02:53:31.766619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27194,"num_bytes_recovered":249951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":494460,"kb_used_data":225244,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188249220,"statfs":{"total":193273528320,"available":192767201280,"internally_reserved":0,"allocated":230649856,"data_stored":730762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":1700,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":53561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002231"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835726+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:26.835726+0000","last_peered":"2024-09-16T02:53:26.835726+0000","last_clean":"2024-09-16T02:53:26.835726+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:26.835726+0000","last_undegraded":"2024-09-16T02:53:26.835726+0000","last_fullsized":"2024-09-16T02:53:26.835726+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4599,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:29.212372+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:29.212372+0000","last_peered":"2024-09-16T02:53:29.212372+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:29.212372+0000","last_undegraded":"2024-09-16T02:53:29.212372+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":165},{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4078,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835101+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:26.835101+0000","last_peered":"2024-09-16T02:53:26.835101+0000","last_clean":"2024-09-16T02:53:26.835101+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:26.835101+0000","last_undegraded":"2024-09-16T02:53:26.835101+0000","last_fullsized":"2024-09-16T02:53:26.835101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27194,"num_bytes_recovered":249951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377704448,"data_stored":812671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247536,"kb_used_data":112224,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94124304,"statfs":{"total":96636764160,"available":96383287296,"internally_reserved":0,"allocated":114917376,"data_stored":362614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673260,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246924,"kb_used_data":113020,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":94124916,"statfs":{"total":96636764160,"available":96383913984,"internally_reserved":0,"allocated":115732480,"data_stored":368148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115486720,"data_stored":250885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114728960,"data_stored":250700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:32.518 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:32.518 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 4599, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:29.212372+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:29.212372+0000', 'last_peered': '2024-09-16T02:53:29.212372+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:29.212372+0000', 'last_undegraded': '2024-09-16T02:53:29.212372+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7718, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 166, 'num_bytes_recovered': 164, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 165}, {'shards': '2,3', 'objects': 3859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:32.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:32.933 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:32.933 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1472,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":506327040,"bytes_avail":192767201280,"bytes_total":193273528320,"misplaced_objects":7718,"misplaced_total":63846,"misplaced_ratio":0.12088462863765936,"recovering_objects_per_sec":145,"recovering_bytes_per_sec":6693,"recovering_keys_per_sec":0,"num_objects_recovered":1162,"num_bytes_recovered":53561,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:32.953 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:32.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:33.257 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:33.257 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:33.276 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":773,"stamp":"2024-09-16T02:53:31.766619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27194,"num_bytes_recovered":249951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":494460,"kb_used_data":225244,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188249220,"statfs":{"total":193273528320,"available":192767201280,"internally_reserved":0,"allocated":230649856,"data_stored":730762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":1700,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1162,"num_bytes_recovered":53561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002231"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835726+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:26.835726+0000","last_peered":"2024-09-16T02:53:26.835726+0000","last_clean":"2024-09-16T02:53:26.835726+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:26.835726+0000","last_undegraded":"2024-09-16T02:53:26.835726+0000","last_fullsized":"2024-09-16T02:53:26.835726+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4599,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:29.212372+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:29.212372+0000","last_peered":"2024-09-16T02:53:29.212372+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:29.212372+0000","last_undegraded":"2024-09-16T02:53:29.212372+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":165},{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4078,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:53:26.835101+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:26.835101+0000","last_peered":"2024-09-16T02:53:26.835101+0000","last_clean":"2024-09-16T02:53:26.835101+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:26.835101+0000","last_undegraded":"2024-09-16T02:53:26.835101+0000","last_fullsized":"2024-09-16T02:53:26.835101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7718,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27194,"num_bytes_recovered":249951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377704448,"data_stored":812671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247536,"kb_used_data":112224,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94124304,"statfs":{"total":96636764160,"available":96383287296,"internally_reserved":0,"allocated":114917376,"data_stored":362614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673260,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246924,"kb_used_data":113020,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":94124916,"statfs":{"total":96636764160,"available":96383913984,"internally_reserved":0,"allocated":115732480,"data_stored":368148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115486720,"data_stored":250885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114728960,"data_stored":250700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:36.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:36.611 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:36.611 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:36.631 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":775,"stamp":"2024-09-16T02:53:35.767896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27430,"num_bytes_recovered":250187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":495404,"kb_used_data":226188,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188248276,"statfs":{"total":193273528320,"available":192766234624,"internally_reserved":0,"allocated":231616512,"data_stored":736333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":2596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":53113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.935804"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":5307,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:34.211705+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:34.211705+0000","last_peered":"2024-09-16T02:53:34.211705+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:34.211705+0000","last_undegraded":"2024-09-16T02:53:34.211705+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":401},{"shards":"2,3","objects":3623}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27430,"num_bytes_recovered":250187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378716160,"data_stored":812918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640556,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248480,"kb_used_data":113168,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94123360,"statfs":{"total":96636764160,"available":96382320640,"internally_reserved":0,"allocated":115884032,"data_stored":368185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673260,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246924,"kb_used_data":113020,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":94124916,"statfs":{"total":96636764160,"available":96383913984,"internally_reserved":0,"allocated":115732480,"data_stored":368148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115486720,"data_stored":250885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115740672,"data_stored":250947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:36.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:36.991 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:37.014 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":775,"stamp":"2024-09-16T02:53:35.767896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27430,"num_bytes_recovered":250187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":495404,"kb_used_data":226188,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188248276,"statfs":{"total":193273528320,"available":192766234624,"internally_reserved":0,"allocated":231616512,"data_stored":736333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":2596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":53113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.935804"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":5307,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:34.211705+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:34.211705+0000","last_peered":"2024-09-16T02:53:34.211705+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:34.211705+0000","last_undegraded":"2024-09-16T02:53:34.211705+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":401},{"shards":"2,3","objects":3623}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27430,"num_bytes_recovered":250187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378716160,"data_stored":812918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640556,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248480,"kb_used_data":113168,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94123360,"statfs":{"total":96636764160,"available":96382320640,"internally_reserved":0,"allocated":115884032,"data_stored":368185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673260,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246924,"kb_used_data":113020,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":94124916,"statfs":{"total":96636764160,"available":96383913984,"internally_reserved":0,"allocated":115732480,"data_stored":368148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115486720,"data_stored":250885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115740672,"data_stored":250947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:37.015 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:37.015 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 5307, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:34.211705+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:34.211705+0000', 'last_peered': '2024-09-16T02:53:34.211705+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:34.211705+0000', 'last_undegraded': '2024-09-16T02:53:34.211705+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7246, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 402, 'num_bytes_recovered': 400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 401}, {'shards': '2,3', 'objects': 3623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:37.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:37.426 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:37.426 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1477,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":507293696,"bytes_avail":192766234624,"bytes_total":193273528320,"misplaced_objects":7246,"misplaced_total":63846,"misplaced_ratio":0.11349183973937287,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":5943,"recovering_keys_per_sec":0,"num_objects_recovered":714,"num_bytes_recovered":53113,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:37.446 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:37.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:37.775 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:37.776 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:37.796 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2024-09-16T02:53:37.768397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27430,"num_bytes_recovered":250187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":480744,"kb_used_data":228168,"kb_used_omap":3,"kb_used_meta":252540,"kb_avail":188262936,"statfs":{"total":193273528320,"available":192781246464,"internally_reserved":0,"allocated":233644032,"data_stored":736828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":258601866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":2596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":53113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002936"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":5307,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:34.211705+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:34.211705+0000","last_peered":"2024-09-16T02:53:34.211705+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:34.211705+0000","last_undegraded":"2024-09-16T02:53:34.211705+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":401},{"shards":"2,3","objects":3623}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7246,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27430,"num_bytes_recovered":250187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380805120,"data_stored":813428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249516,"kb_used_data":114204,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94122324,"statfs":{"total":96636764160,"available":96381259776,"internally_reserved":0,"allocated":116944896,"data_stored":368444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673261,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231228,"kb_used_data":113964,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94140612,"statfs":{"total":96636764160,"available":96399986688,"internally_reserved":0,"allocated":116699136,"data_stored":368384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986249,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":116502528,"data_stored":251133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116813824,"data_stored":251209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:40.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:41.147 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:41.148 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:41.169 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":777,"stamp":"2024-09-16T02:53:39.768847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":480744,"kb_used_data":228168,"kb_used_omap":3,"kb_used_meta":252540,"kb_avail":188262936,"statfs":{"total":193273528320,"available":192781246464,"internally_reserved":0,"allocated":233644032,"data_stored":736828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":258601866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":2104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":960,"num_bytes_recovered":53359,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.917956"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":6046,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:39.223391+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:39.223391+0000","last_peered":"2024-09-16T02:53:39.223391+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:39.223391+0000","last_undegraded":"2024-09-16T02:53:39.223391+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":647},{"shards":"2,3","objects":3377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380805120,"data_stored":813428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249516,"kb_used_data":114204,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94122324,"statfs":{"total":96636764160,"available":96381259776,"internally_reserved":0,"allocated":116944896,"data_stored":368444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673261,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231228,"kb_used_data":113964,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94140612,"statfs":{"total":96636764160,"available":96399986688,"internally_reserved":0,"allocated":116699136,"data_stored":368384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986249,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":116502528,"data_stored":251133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116813824,"data_stored":251209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:41.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:41.501 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:41.502 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:41.522 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":777,"stamp":"2024-09-16T02:53:39.768847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":480744,"kb_used_data":228168,"kb_used_omap":3,"kb_used_meta":252540,"kb_avail":188262936,"statfs":{"total":193273528320,"available":192781246464,"internally_reserved":0,"allocated":233644032,"data_stored":736828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":258601866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4024,"num_objects_misplaced":2104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":960,"num_bytes_recovered":53359,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.917956"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4064,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865904+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865904+0000","last_peered":"2024-09-16T02:53:27.865904+0000","last_clean":"2024-09-16T02:53:27.865904+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865904+0000","last_undegraded":"2024-09-16T02:53:27.865904+0000","last_fullsized":"2024-09-16T02:53:27.865904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862497+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862497+0000","last_peered":"2024-09-16T02:53:27.862497+0000","last_clean":"2024-09-16T02:53:27.862497+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862497+0000","last_undegraded":"2024-09-16T02:53:27.862497+0000","last_fullsized":"2024-09-16T02:53:27.862497+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":6046,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:39.223391+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:39.223391+0000","last_peered":"2024-09-16T02:53:39.223391+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:39.223391+0000","last_undegraded":"2024-09-16T02:53:39.223391+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":647},{"shards":"2,3","objects":3377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380805120,"data_stored":813428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249516,"kb_used_data":114204,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94122324,"statfs":{"total":96636764160,"available":96381259776,"internally_reserved":0,"allocated":116944896,"data_stored":368444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673261,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":231228,"kb_used_data":113964,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94140612,"statfs":{"total":96636764160,"available":96399986688,"internally_reserved":0,"allocated":116699136,"data_stored":368384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986249,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":116502528,"data_stored":251133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116813824,"data_stored":251209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:41.523 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:41.523 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 6046, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:39.223391+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:39.223391+0000', 'last_peered': '2024-09-16T02:53:39.223391+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:39.223391+0000', 'last_undegraded': '2024-09-16T02:53:39.223391+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6754, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 648, 'num_bytes_recovered': 646, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 647}, {'shards': '2,3', 'objects': 3377}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:41.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:41.926 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:41.926 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1481,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":493305856,"bytes_avail":192780222464,"bytes_total":193273528320,"misplaced_objects":6754,"misplaced_total":63846,"misplaced_ratio":0.10578579707420982,"recovering_objects_per_sec":66,"recovering_bytes_per_sec":4431,"recovering_keys_per_sec":0,"num_objects_recovered":795,"num_bytes_recovered":53195,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:41.949 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:41.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:42.267 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:42.267 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:42.289 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":778,"stamp":"2024-09-16T02:53:41.769447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":481744,"kb_used_data":229168,"kb_used_omap":3,"kb_used_meta":252540,"kb_avail":188261936,"statfs":{"total":193273528320,"available":192780222464,"internally_reserved":0,"allocated":234668032,"data_stored":737078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":258601866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1590,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":53195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":6046,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:39.223391+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:39.223391+0000","last_peered":"2024-09-16T02:53:39.223391+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:39.223391+0000","last_undegraded":"2024-09-16T02:53:39.223391+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":647},{"shards":"2,3","objects":3377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381906944,"data_stored":813697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249516,"kb_used_data":114204,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94122324,"statfs":{"total":96636764160,"available":96381259776,"internally_reserved":0,"allocated":116944896,"data_stored":368444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232228,"kb_used_data":114964,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94139612,"statfs":{"total":96636764160,"available":96398962688,"internally_reserved":0,"allocated":117723136,"data_stored":368634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986250,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117604352,"data_stored":251402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116813824,"data_stored":251209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:45.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:45.633 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:45.633 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:45.655 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":779,"stamp":"2024-09-16T02:53:43.770001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":486968,"kb_used_data":230232,"kb_used_omap":3,"kb_used_meta":256700,"kb_avail":188256712,"statfs":{"total":193273528320,"available":192774873088,"internally_reserved":0,"allocated":235757568,"data_stored":737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":262861706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-964,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":482,"num_bytes_recovered":482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003382"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":6046,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:39.223391+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:39.223391+0000","last_peered":"2024-09-16T02:53:39.223391+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:39.223391+0000","last_undegraded":"2024-09-16T02:53:39.223391+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":647},{"shards":"2,3","objects":3377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6754,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27676,"num_bytes_recovered":250433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":383025152,"data_stored":813970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640558,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254740,"kb_used_data":115268,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94117100,"statfs":{"total":96636764160,"available":96375910400,"internally_reserved":0,"allocated":118034432,"data_stored":368710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232228,"kb_used_data":114964,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94139612,"statfs":{"total":96636764160,"available":96398962688,"internally_reserved":0,"allocated":117723136,"data_stored":368634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986250,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117604352,"data_stored":251402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117932032,"data_stored":251482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:45.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:46.005 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:46.005 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:46.025 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":780,"stamp":"2024-09-16T02:53:45.770593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6228,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27939,"num_bytes_recovered":250696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":486968,"kb_used_data":230232,"kb_used_omap":3,"kb_used_meta":256700,"kb_avail":188256712,"statfs":{"total":193273528320,"available":192774873088,"internally_reserved":0,"allocated":235757568,"data_stored":737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":262861706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1490,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003284"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":6835,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:44.231205+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:44.231205+0000","last_peered":"2024-09-16T02:53:44.231205+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:44.231205+0000","last_undegraded":"2024-09-16T02:53:44.231205+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6228,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":909,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":910},{"shards":"2,3","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6228,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27939,"num_bytes_recovered":250696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":383025152,"data_stored":813970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640558,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254740,"kb_used_data":115268,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94117100,"statfs":{"total":96636764160,"available":96375910400,"internally_reserved":0,"allocated":118034432,"data_stored":368710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232228,"kb_used_data":114964,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94139612,"statfs":{"total":96636764160,"available":96398962688,"internally_reserved":0,"allocated":117723136,"data_stored":368634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986250,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117604352,"data_stored":251402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117932032,"data_stored":251482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:46.026 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:46.026 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 6835, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:44.231205+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:44.231205+0000', 'last_peered': '2024-09-16T02:53:44.231205+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:44.231205+0000', 'last_undegraded': '2024-09-16T02:53:44.231205+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6228, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 911, 'num_bytes_recovered': 909, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 910}, {'shards': '2,3', 'objects': 3114}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:46.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:46.485 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1486,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":498655232,"bytes_avail":192774873088,"bytes_total":193273528320,"misplaced_objects":6228,"misplaced_total":63846,"misplaced_ratio":0.097547223005356634,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":62,"recovering_keys_per_sec":0,"num_objects_recovered":745,"num_bytes_recovered":745,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:46.505 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:46.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:46.835 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:46.836 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:46.855 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":780,"stamp":"2024-09-16T02:53:45.770593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6228,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27939,"num_bytes_recovered":250696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":486968,"kb_used_data":230232,"kb_used_omap":3,"kb_used_meta":256700,"kb_avail":188256712,"statfs":{"total":193273528320,"available":192774873088,"internally_reserved":0,"allocated":235757568,"data_stored":737344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":262861706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1490,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":745,"num_bytes_recovered":745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003284"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":6835,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:44.231205+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:44.231205+0000","last_peered":"2024-09-16T02:53:44.231205+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:44.231205+0000","last_undegraded":"2024-09-16T02:53:44.231205+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6228,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":909,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":910},{"shards":"2,3","objects":3114}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6228,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27939,"num_bytes_recovered":250696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":383025152,"data_stored":813970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640558,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254740,"kb_used_data":115268,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94117100,"statfs":{"total":96636764160,"available":96375910400,"internally_reserved":0,"allocated":118034432,"data_stored":368710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232228,"kb_used_data":114964,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94139612,"statfs":{"total":96636764160,"available":96398962688,"internally_reserved":0,"allocated":117723136,"data_stored":368634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986250,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":117604352,"data_stored":251402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117932032,"data_stored":251482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:49.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:50.196 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:50.196 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:50.215 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-09-16T02:53:49.771482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":489512,"kb_used_data":232776,"kb_used_omap":3,"kb_used_meta":256700,"kb_avail":188254168,"statfs":{"total":193273528320,"available":192772268032,"internally_reserved":0,"allocated":238362624,"data_stored":737980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":262861706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1560,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":7648,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:49.245334+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:49.245334+0000","last_peered":"2024-09-16T02:53:49.245334+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:49.245334+0000","last_undegraded":"2024-09-16T02:53:49.245334+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":1180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1181},{"shards":"2,3","objects":2843}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385290240,"data_stored":814523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640559,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255904,"kb_used_data":116432,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94115936,"statfs":{"total":96636764160,"available":96374718464,"internally_reserved":0,"allocated":119226368,"data_stored":369001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673263,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233608,"kb_used_data":116344,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94138232,"statfs":{"total":96636764160,"available":96397549568,"internally_reserved":0,"allocated":119136256,"data_stored":368979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986251,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118775808,"data_stored":251688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119025664,"data_stored":251749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:50.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:50.545 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:50.545 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:50.566 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-09-16T02:53:49.771482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":489512,"kb_used_data":232776,"kb_used_omap":3,"kb_used_meta":256700,"kb_avail":188254168,"statfs":{"total":193273528320,"available":192772268032,"internally_reserved":0,"allocated":238362624,"data_stored":737980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":262861706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1560,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":7648,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:49.245334+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:49.245334+0000","last_peered":"2024-09-16T02:53:49.245334+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:49.245334+0000","last_undegraded":"2024-09-16T02:53:49.245334+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":1180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1181},{"shards":"2,3","objects":2843}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385290240,"data_stored":814523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640559,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255904,"kb_used_data":116432,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94115936,"statfs":{"total":96636764160,"available":96374718464,"internally_reserved":0,"allocated":119226368,"data_stored":369001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673263,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233608,"kb_used_data":116344,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94138232,"statfs":{"total":96636764160,"available":96397549568,"internally_reserved":0,"allocated":119136256,"data_stored":368979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986251,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118775808,"data_stored":251688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119025664,"data_stored":251749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:50.567 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:50.567 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 7648, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:49.245334+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:49.245334+0000', 'last_peered': '2024-09-16T02:53:49.245334+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:49.245334+0000', 'last_undegraded': '2024-09-16T02:53:49.245334+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5686, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1182, 'num_bytes_recovered': 1180, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1181}, {'shards': '2,3', 'objects': 2843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:50.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:50.999 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:50.999 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1490,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":501260288,"bytes_avail":192772268032,"bytes_total":193273528320,"misplaced_objects":5686,"misplaced_total":63846,"misplaced_ratio":0.089058045923002205,"recovering_objects_per_sec":64,"recovering_bytes_per_sec":64,"recovering_keys_per_sec":0,"num_objects_recovered":780,"num_bytes_recovered":780,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:51.018 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:51.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:51.368 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:51.368 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:51.387 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-09-16T02:53:49.771482+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":489512,"kb_used_data":232776,"kb_used_omap":3,"kb_used_meta":256700,"kb_avail":188254168,"statfs":{"total":193273528320,"available":192772268032,"internally_reserved":0,"allocated":238362624,"data_stored":737980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":262861706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1560,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":7648,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:49.245334+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:49.245334+0000","last_peered":"2024-09-16T02:53:49.245334+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:49.245334+0000","last_undegraded":"2024-09-16T02:53:49.245334+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":1180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1181},{"shards":"2,3","objects":2843}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385290240,"data_stored":814523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640559,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":255904,"kb_used_data":116432,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94115936,"statfs":{"total":96636764160,"available":96374718464,"internally_reserved":0,"allocated":119226368,"data_stored":369001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673263,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233608,"kb_used_data":116344,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":94138232,"statfs":{"total":96636764160,"available":96397549568,"internally_reserved":0,"allocated":119136256,"data_stored":368979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986251,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":118775808,"data_stored":251688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119025664,"data_stored":251749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:54.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:54.734 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:54.735 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:54.755 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":784,"stamp":"2024-09-16T02:53:53.772930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":495772,"kb_used_data":234876,"kb_used_omap":3,"kb_used_meta":260860,"kb_avail":188247908,"statfs":{"total":193273528320,"available":192765857792,"internally_reserved":0,"allocated":240513024,"data_stored":738505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":267121546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1068,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003483"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":7648,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:49.245334+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:49.245334+0000","last_peered":"2024-09-16T02:53:49.245334+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:49.245334+0000","last_undegraded":"2024-09-16T02:53:49.245334+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":1180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1181},{"shards":"2,3","objects":2843}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":387719168,"data_stored":815116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640560,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257088,"kb_used_data":117616,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94114752,"statfs":{"total":96636764160,"available":96373506048,"internally_reserved":0,"allocated":120438784,"data_stored":369297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673264,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238684,"kb_used_data":117260,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94133156,"statfs":{"total":96636764160,"available":96392351744,"internally_reserved":0,"allocated":120074240,"data_stored":369208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986252,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120025088,"data_stored":251993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120205312,"data_stored":252037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:54.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:55.085 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:55.105 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":784,"stamp":"2024-09-16T02:53:53.772930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":495772,"kb_used_data":234876,"kb_used_omap":3,"kb_used_meta":260860,"kb_avail":188247908,"statfs":{"total":193273528320,"available":192765857792,"internally_reserved":0,"allocated":240513024,"data_stored":738505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":267121546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1068,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003483"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":7648,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:49.245334+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:49.245334+0000","last_peered":"2024-09-16T02:53:49.245334+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:49.245334+0000","last_undegraded":"2024-09-16T02:53:49.245334+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":1180,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1181},{"shards":"2,3","objects":2843}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5686,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28210,"num_bytes_recovered":250967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":387719168,"data_stored":815116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640560,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257088,"kb_used_data":117616,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94114752,"statfs":{"total":96636764160,"available":96373506048,"internally_reserved":0,"allocated":120438784,"data_stored":369297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673264,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238684,"kb_used_data":117260,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94133156,"statfs":{"total":96636764160,"available":96392351744,"internally_reserved":0,"allocated":120074240,"data_stored":369208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986252,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120025088,"data_stored":251993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120205312,"data_stored":252037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:55.106 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:55.106 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 7648, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:49.245334+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:49.245334+0000', 'last_peered': '2024-09-16T02:53:49.245334+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:49.245334+0000', 'last_undegraded': '2024-09-16T02:53:49.245334+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5686, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1182, 'num_bytes_recovered': 1180, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1181}, {'shards': '2,3', 'objects': 2843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:55.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:53:55.560 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:55.560 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1495,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":507670528,"bytes_avail":192765857792,"bytes_total":193273528320,"misplaced_objects":5686,"misplaced_total":63846,"misplaced_ratio":0.089058045923002205,"recovering_objects_per_sec":44,"recovering_bytes_per_sec":44,"recovering_keys_per_sec":0,"num_objects_recovered":534,"num_bytes_recovered":534,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:53:55.580 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:53:55.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:55.904 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:55.905 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:55.924 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":785,"stamp":"2024-09-16T02:53:55.773603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28506,"num_bytes_recovered":251263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":495772,"kb_used_data":234876,"kb_used_omap":3,"kb_used_meta":260860,"kb_avail":188247908,"statfs":{"total":193273528320,"available":192765857792,"internally_reserved":0,"allocated":240513024,"data_stored":738505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":267121546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1660,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":830,"num_bytes_recovered":830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003601"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":8536,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:54.254908+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:54.254908+0000","last_peered":"2024-09-16T02:53:54.254908+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:54.254908+0000","last_undegraded":"2024-09-16T02:53:54.254908+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1477},{"shards":"2,3","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28506,"num_bytes_recovered":251263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":387719168,"data_stored":815116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640560,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257088,"kb_used_data":117616,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94114752,"statfs":{"total":96636764160,"available":96373506048,"internally_reserved":0,"allocated":120438784,"data_stored":369297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673264,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":238684,"kb_used_data":117260,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94133156,"statfs":{"total":96636764160,"available":96392351744,"internally_reserved":0,"allocated":120074240,"data_stored":369208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986252,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":120025088,"data_stored":251993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120205312,"data_stored":252037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:58.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:59.282 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:59.283 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:59.303 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":786,"stamp":"2024-09-16T02:53:57.773968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28506,"num_bytes_recovered":251263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":498284,"kb_used_data":237388,"kb_used_omap":3,"kb_used_meta":260860,"kb_avail":188245396,"statfs":{"total":193273528320,"available":192763285504,"internally_reserved":0,"allocated":243085312,"data_stored":739133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":267121546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1134,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003375"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":8536,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:54.254908+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:54.254908+0000","last_peered":"2024-09-16T02:53:54.254908+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:54.254908+0000","last_undegraded":"2024-09-16T02:53:54.254908+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1477},{"shards":"2,3","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28506,"num_bytes_recovered":251263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":390230016,"data_stored":815729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640561,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258312,"kb_used_data":118840,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94113528,"statfs":{"total":96636764160,"available":96372252672,"internally_reserved":0,"allocated":121692160,"data_stored":369603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673265,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":239972,"kb_used_data":118548,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94131868,"statfs":{"total":96636764160,"available":96391032832,"internally_reserved":0,"allocated":121393152,"data_stored":369530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986253,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121270272,"data_stored":252297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121470976,"data_stored":252346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:59.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:53:59.637 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:53:59.638 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:53:59.658 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":786,"stamp":"2024-09-16T02:53:57.773968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28506,"num_bytes_recovered":251263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":498284,"kb_used_data":237388,"kb_used_omap":3,"kb_used_meta":260860,"kb_avail":188245396,"statfs":{"total":193273528320,"available":192763285504,"internally_reserved":0,"allocated":243085312,"data_stored":739133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":267121546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1134,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":567,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003375"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":8536,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:54.254908+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:54.254908+0000","last_peered":"2024-09-16T02:53:54.254908+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:54.254908+0000","last_undegraded":"2024-09-16T02:53:54.254908+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1477},{"shards":"2,3","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5094,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28506,"num_bytes_recovered":251263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":390230016,"data_stored":815729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640561,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258312,"kb_used_data":118840,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94113528,"statfs":{"total":96636764160,"available":96372252672,"internally_reserved":0,"allocated":121692160,"data_stored":369603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673265,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":239972,"kb_used_data":118548,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94131868,"statfs":{"total":96636764160,"available":96391032832,"internally_reserved":0,"allocated":121393152,"data_stored":369530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986253,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121270272,"data_stored":252297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121470976,"data_stored":252346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:53:59.659 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:53:59.659 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 8536, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:54.254908+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:54.254908+0000', 'last_peered': '2024-09-16T02:53:54.254908+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:54.254908+0000', 'last_undegraded': '2024-09-16T02:53:54.254908+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5094, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1478, 'num_bytes_recovered': 1476, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1477}, {'shards': '2,3', 'objects': 2547}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:53:59.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:54:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:00.059 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1499,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":510242816,"bytes_avail":192763285504,"bytes_total":193273528320,"misplaced_objects":4444,"misplaced_total":63846,"misplaced_ratio":0.069604986999968671,"recovering_objects_per_sec":74,"recovering_bytes_per_sec":74,"recovering_keys_per_sec":0,"num_objects_recovered":892,"num_bytes_recovered":892,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:54:00.080 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:54:00.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:00.418 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:00.418 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:00.437 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":787,"stamp":"2024-09-16T02:53:59.774501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":498284,"kb_used_data":237388,"kb_used_omap":3,"kb_used_meta":260860,"kb_avail":188245396,"statfs":{"total":193273528320,"available":192763285504,"internally_reserved":0,"allocated":243085312,"data_stored":739133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":267121546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1784,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":892,"num_bytes_recovered":892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003586"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":9511,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:59.261265+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:59.261265+0000","last_peered":"2024-09-16T02:53:59.261265+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:59.261265+0000","last_undegraded":"2024-09-16T02:53:59.261265+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1803,"num_bytes_recovered":1801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1802},{"shards":"2,3","objects":2222}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":390230016,"data_stored":815729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640561,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258312,"kb_used_data":118840,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94113528,"statfs":{"total":96636764160,"available":96372252672,"internally_reserved":0,"allocated":121692160,"data_stored":369603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673265,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":239972,"kb_used_data":118548,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94131868,"statfs":{"total":96636764160,"available":96391032832,"internally_reserved":0,"allocated":121393152,"data_stored":369530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986253,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121270272,"data_stored":252297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121470976,"data_stored":252346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:03.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:03.811 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:03.812 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:03.833 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-09-16T02:54:03.775430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":505188,"kb_used_data":240132,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188238492,"statfs":{"total":193273528320,"available":192756215808,"internally_reserved":0,"allocated":245895168,"data_stored":739819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1242,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003204"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":9511,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:59.261265+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:59.261265+0000","last_peered":"2024-09-16T02:53:59.261265+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:59.261265+0000","last_undegraded":"2024-09-16T02:53:59.261265+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1803,"num_bytes_recovered":1801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1802},{"shards":"2,3","objects":2222}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393043968,"data_stored":816416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640562,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263852,"kb_used_data":120220,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94107988,"statfs":{"total":96636764160,"available":96366579712,"internally_reserved":0,"allocated":123105280,"data_stored":369948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241336,"kb_used_data":119912,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94130504,"statfs":{"total":96636764160,"available":96389636096,"internally_reserved":0,"allocated":122789888,"data_stored":369871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986254,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":122671104,"data_stored":252639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122884096,"data_stored":252691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:03.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:04.159 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:04.159 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:04.180 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-09-16T02:54:03.775430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":505188,"kb_used_data":240132,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188238492,"statfs":{"total":193273528320,"available":192756215808,"internally_reserved":0,"allocated":245895168,"data_stored":739819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1242,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003204"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":9511,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:59.261265+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:59.261265+0000","last_peered":"2024-09-16T02:53:59.261265+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:59.261265+0000","last_undegraded":"2024-09-16T02:53:59.261265+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1803,"num_bytes_recovered":1801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1802},{"shards":"2,3","objects":2222}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393043968,"data_stored":816416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640562,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263852,"kb_used_data":120220,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94107988,"statfs":{"total":96636764160,"available":96366579712,"internally_reserved":0,"allocated":123105280,"data_stored":369948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241336,"kb_used_data":119912,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94130504,"statfs":{"total":96636764160,"available":96389636096,"internally_reserved":0,"allocated":122789888,"data_stored":369871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986254,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":122671104,"data_stored":252639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122884096,"data_stored":252691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:04.180 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:54:04.181 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 9511, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:53:59.261265+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:53:59.261265+0000', 'last_peered': '2024-09-16T02:53:59.261265+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:53:59.261265+0000', 'last_undegraded': '2024-09-16T02:53:59.261265+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4444, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1803, 'num_bytes_recovered': 1801, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1802}, {'shards': '2,3', 'objects': 2222}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:54:04.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:54:04.587 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:04.587 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1504,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":517312512,"bytes_avail":192756215808,"bytes_total":193273528320,"misplaced_objects":4444,"misplaced_total":63846,"misplaced_ratio":0.069604986999968671,"recovering_objects_per_sec":51,"recovering_bytes_per_sec":51,"recovering_keys_per_sec":0,"num_objects_recovered":621,"num_bytes_recovered":621,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:54:04.605 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:54:04.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:04.927 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:04.948 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-09-16T02:54:03.775430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":505188,"kb_used_data":240132,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188238492,"statfs":{"total":193273528320,"available":192756215808,"internally_reserved":0,"allocated":245895168,"data_stored":739819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1242,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":621,"num_bytes_recovered":621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003204"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4007,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860396+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860396+0000","last_peered":"2024-09-16T02:53:27.860396+0000","last_clean":"2024-09-16T02:53:27.860396+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860396+0000","last_undegraded":"2024-09-16T02:53:27.860396+0000","last_fullsized":"2024-09-16T02:53:27.860396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":9511,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:53:59.261265+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:53:59.261265+0000","last_peered":"2024-09-16T02:53:59.261265+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:53:59.261265+0000","last_undegraded":"2024-09-16T02:53:59.261265+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1803,"num_bytes_recovered":1801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1802},{"shards":"2,3","objects":2222}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4444,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28831,"num_bytes_recovered":251588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393043968,"data_stored":816416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640562,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263852,"kb_used_data":120220,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94107988,"statfs":{"total":96636764160,"available":96366579712,"internally_reserved":0,"allocated":123105280,"data_stored":369948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":241336,"kb_used_data":119912,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94130504,"statfs":{"total":96636764160,"available":96389636096,"internally_reserved":0,"allocated":122789888,"data_stored":369871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986254,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":122671104,"data_stored":252639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122884096,"data_stored":252691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:07.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:08.317 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:08.340 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":791,"stamp":"2024-09-16T02:54:07.776286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29218,"num_bytes_recovered":251975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":512356,"kb_used_data":243140,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188231324,"statfs":{"total":193273528320,"available":192748875776,"internally_reserved":0,"allocated":248975360,"data_stored":740571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1424,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002682"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":10672,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:04.266086+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:04.266086+0000","last_peered":"2024-09-16T02:54:04.266086+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:04.266086+0000","last_undegraded":"2024-09-16T02:54:04.266086+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2190,"num_bytes_recovered":2188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2189},{"shards":"2,3","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29218,"num_bytes_recovered":251975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":396111872,"data_stored":817165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640563,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265340,"kb_used_data":121708,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94106500,"statfs":{"total":96636764160,"available":96365056000,"internally_reserved":0,"allocated":124628992,"data_stored":370320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247016,"kb_used_data":121432,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94124824,"statfs":{"total":96636764160,"available":96383819776,"internally_reserved":0,"allocated":124346368,"data_stored":370251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986255,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124190720,"data_stored":253010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124432384,"data_stored":253069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:08.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:08.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:08.672 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:08.693 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":791,"stamp":"2024-09-16T02:54:07.776286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29218,"num_bytes_recovered":251975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":512356,"kb_used_data":243140,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188231324,"statfs":{"total":193273528320,"available":192748875776,"internally_reserved":0,"allocated":248975360,"data_stored":740571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1424,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002682"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":10672,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:04.266086+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:04.266086+0000","last_peered":"2024-09-16T02:54:04.266086+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:04.266086+0000","last_undegraded":"2024-09-16T02:54:04.266086+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2190,"num_bytes_recovered":2188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2189},{"shards":"2,3","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29218,"num_bytes_recovered":251975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":396111872,"data_stored":817165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640563,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265340,"kb_used_data":121708,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94106500,"statfs":{"total":96636764160,"available":96365056000,"internally_reserved":0,"allocated":124628992,"data_stored":370320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247016,"kb_used_data":121432,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94124824,"statfs":{"total":96636764160,"available":96383819776,"internally_reserved":0,"allocated":124346368,"data_stored":370251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986255,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124190720,"data_stored":253010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124432384,"data_stored":253069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:08.693 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:54:08.694 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 10672, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:54:04.266086+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:54:04.266086+0000', 'last_peered': '2024-09-16T02:54:04.266086+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:54:04.266086+0000', 'last_undegraded': '2024-09-16T02:54:04.266086+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3670, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2190, 'num_bytes_recovered': 2188, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2189}, {'shards': '2,3', 'objects': 1835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:54:08.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:54:09.110 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:09.110 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1508,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":524652544,"bytes_avail":192748875776,"bytes_total":193273528320,"misplaced_objects":3670,"misplaced_total":63846,"misplaced_ratio":0.057482066221846317,"recovering_objects_per_sec":59,"recovering_bytes_per_sec":59,"recovering_keys_per_sec":0,"num_objects_recovered":712,"num_bytes_recovered":712,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:54:09.130 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:54:09.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:09.457 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:09.457 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:09.475 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":791,"stamp":"2024-09-16T02:54:07.776286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29218,"num_bytes_recovered":251975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":512356,"kb_used_data":243140,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188231324,"statfs":{"total":193273528320,"available":192748875776,"internally_reserved":0,"allocated":248975360,"data_stored":740571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1424,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002682"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":10672,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:04.266086+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:04.266086+0000","last_peered":"2024-09-16T02:54:04.266086+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:04.266086+0000","last_undegraded":"2024-09-16T02:54:04.266086+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2190,"num_bytes_recovered":2188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2189},{"shards":"2,3","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3670,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29218,"num_bytes_recovered":251975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":396111872,"data_stored":817165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640563,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265340,"kb_used_data":121708,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94106500,"statfs":{"total":96636764160,"available":96365056000,"internally_reserved":0,"allocated":124628992,"data_stored":370320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673267,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":247016,"kb_used_data":121432,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94124824,"statfs":{"total":96636764160,"available":96383819776,"internally_reserved":0,"allocated":124346368,"data_stored":370251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986255,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":124190720,"data_stored":253010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124432384,"data_stored":253069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:12.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:12.830 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:12.850 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":793,"stamp":"2024-09-16T02:54:11.777389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29647,"num_bytes_recovered":252404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":514052,"kb_used_data":244836,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188229628,"statfs":{"total":193273528320,"available":192747139072,"internally_reserved":0,"allocated":250712064,"data_stored":740995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1632,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":11959,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:09.267626+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:09.267626+0000","last_peered":"2024-09-16T02:54:09.267626+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:09.267626+0000","last_undegraded":"2024-09-16T02:54:09.267626+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2619,"num_bytes_recovered":2617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2618},{"shards":"2,3","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29647,"num_bytes_recovered":252404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397860864,"data_stored":817592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640563,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265340,"kb_used_data":121708,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94106500,"statfs":{"total":96636764160,"available":96365056000,"internally_reserved":0,"allocated":124628992,"data_stored":370320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248712,"kb_used_data":123128,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94123128,"statfs":{"total":96636764160,"available":96382083072,"internally_reserved":0,"allocated":126083072,"data_stored":370675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986256,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125939712,"data_stored":253437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124432384,"data_stored":253069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:12.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:13.182 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:13.182 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:13.203 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":793,"stamp":"2024-09-16T02:54:11.777389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29647,"num_bytes_recovered":252404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":514052,"kb_used_data":244836,"kb_used_omap":3,"kb_used_meta":269180,"kb_avail":188229628,"statfs":{"total":193273528320,"available":192747139072,"internally_reserved":0,"allocated":250712064,"data_stored":740995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":275641226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1632,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":11959,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:09.267626+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:09.267626+0000","last_peered":"2024-09-16T02:54:09.267626+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:09.267626+0000","last_undegraded":"2024-09-16T02:54:09.267626+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2619,"num_bytes_recovered":2617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2618},{"shards":"2,3","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29647,"num_bytes_recovered":252404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397860864,"data_stored":817592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640563,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265340,"kb_used_data":121708,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94106500,"statfs":{"total":96636764160,"available":96365056000,"internally_reserved":0,"allocated":124628992,"data_stored":370320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248712,"kb_used_data":123128,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94123128,"statfs":{"total":96636764160,"available":96382083072,"internally_reserved":0,"allocated":126083072,"data_stored":370675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986256,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125939712,"data_stored":253437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124432384,"data_stored":253069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:13.203 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:54:13.204 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 11959, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:54:09.267626+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:54:09.267626+0000', 'last_peered': '2024-09-16T02:54:09.267626+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:54:09.267626+0000', 'last_undegraded': '2024-09-16T02:54:09.267626+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2812, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2619, 'num_bytes_recovered': 2617, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2618}, {'shards': '2,3', 'objects': 1406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:54:13.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:54:13.610 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:13.610 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1513,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":526389248,"bytes_avail":192747139072,"bytes_total":193273528320,"misplaced_objects":2812,"misplaced_total":63846,"misplaced_ratio":0.044043479622842469,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":67,"recovering_keys_per_sec":0,"num_objects_recovered":816,"num_bytes_recovered":816,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:54:13.629 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:54:13.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:13.967 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:13.968 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:13.986 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-09-16T02:54:13.778067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29647,"num_bytes_recovered":252404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":519944,"kb_used_data":246568,"kb_used_omap":3,"kb_used_meta":273340,"kb_avail":188223736,"statfs":{"total":193273528320,"available":192741105664,"internally_reserved":0,"allocated":252485632,"data_stored":741428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":279901066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1632,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003169"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":11959,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:09.267626+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:09.267626+0000","last_peered":"2024-09-16T02:54:09.267626+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:09.267626+0000","last_undegraded":"2024-09-16T02:54:09.267626+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2619,"num_bytes_recovered":2617,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2618},{"shards":"2,3","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2812,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29647,"num_bytes_recovered":252404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399667200,"data_stored":818033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640564,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271232,"kb_used_data":123440,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94100608,"statfs":{"total":96636764160,"available":96359022592,"internally_reserved":0,"allocated":126402560,"data_stored":370753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248712,"kb_used_data":123128,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94123128,"statfs":{"total":96636764160,"available":96382083072,"internally_reserved":0,"allocated":126083072,"data_stored":370675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986256,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125939712,"data_stored":253437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126238720,"data_stored":253510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:16.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:17.355 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:17.355 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:17.377 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":795,"stamp":"2024-09-16T02:54:15.778486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30151,"num_bytes_recovered":252908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":519944,"kb_used_data":246568,"kb_used_omap":3,"kb_used_meta":273340,"kb_avail":188223736,"statfs":{"total":193273528320,"available":192741105664,"internally_reserved":0,"allocated":252485632,"data_stored":741428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":279901066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2640,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003056"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":13471,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:14.263229+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:14.263229+0000","last_peered":"2024-09-16T02:54:14.263229+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:14.263229+0000","last_undegraded":"2024-09-16T02:54:14.263229+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":3121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3122},{"shards":"2,3","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30151,"num_bytes_recovered":252908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399667200,"data_stored":818033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640564,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271232,"kb_used_data":123440,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94100608,"statfs":{"total":96636764160,"available":96359022592,"internally_reserved":0,"allocated":126402560,"data_stored":370753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248712,"kb_used_data":123128,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94123128,"statfs":{"total":96636764160,"available":96382083072,"internally_reserved":0,"allocated":126083072,"data_stored":370675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986256,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125939712,"data_stored":253437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126238720,"data_stored":253510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:17.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:17.717 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:17.739 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":795,"stamp":"2024-09-16T02:54:15.778486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30151,"num_bytes_recovered":252908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":519944,"kb_used_data":246568,"kb_used_omap":3,"kb_used_meta":273340,"kb_avail":188223736,"statfs":{"total":193273528320,"available":192741105664,"internally_reserved":0,"allocated":252485632,"data_stored":741428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":279901066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2640,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003056"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":13471,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:14.263229+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:14.263229+0000","last_peered":"2024-09-16T02:54:14.263229+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:14.263229+0000","last_undegraded":"2024-09-16T02:54:14.263229+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":3121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3122},{"shards":"2,3","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30151,"num_bytes_recovered":252908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399667200,"data_stored":818033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640564,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":271232,"kb_used_data":123440,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94100608,"statfs":{"total":96636764160,"available":96359022592,"internally_reserved":0,"allocated":126402560,"data_stored":370753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673268,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248712,"kb_used_data":123128,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94123128,"statfs":{"total":96636764160,"available":96382083072,"internally_reserved":0,"allocated":126083072,"data_stored":370675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986256,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125939712,"data_stored":253437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126238720,"data_stored":253510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:17.740 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:54:17.740 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 13471, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:54:14.263229+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:54:14.263229+0000', 'last_peered': '2024-09-16T02:54:14.263229+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:54:14.263229+0000', 'last_undegraded': '2024-09-16T02:54:14.263229+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1804, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3123, 'num_bytes_recovered': 3121, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3122}, {'shards': '2,3', 'objects': 902}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:54:17.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:54:18.169 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:18.169 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1517,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":540962816,"bytes_avail":192732565504,"bytes_total":193273528320,"misplaced_objects":1804,"misplaced_total":63846,"misplaced_ratio":0.028255489772264511,"recovering_objects_per_sec":77,"recovering_bytes_per_sec":77,"recovering_keys_per_sec":0,"num_objects_recovered":933,"num_bytes_recovered":933,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:54:18.191 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:54:18.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:18.533 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:18.533 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:18.554 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":796,"stamp":"2024-09-16T02:54:17.779100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30151,"num_bytes_recovered":252908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":528284,"kb_used_data":250748,"kb_used_omap":3,"kb_used_meta":277500,"kb_avail":188215396,"statfs":{"total":193273528320,"available":192732565504,"internally_reserved":0,"allocated":256765952,"data_stored":742473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":284160906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1866,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":933,"num_bytes_recovered":933,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4065,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.865983+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:53:27.865983+0000","last_peered":"2024-09-16T02:53:27.865983+0000","last_clean":"2024-09-16T02:53:27.865983+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:53:27.865983+0000","last_undegraded":"2024-09-16T02:53:27.865983+0000","last_fullsized":"2024-09-16T02:53:27.865983+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4124,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862572+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:53:27.862572+0000","last_peered":"2024-09-16T02:53:27.862572+0000","last_clean":"2024-09-16T02:53:27.862572+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:53:27.862572+0000","last_undegraded":"2024-09-16T02:53:27.862572+0000","last_fullsized":"2024-09-16T02:53:27.862572+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4015,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.871856+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:53:27.871856+0000","last_peered":"2024-09-16T02:53:27.871856+0000","last_clean":"2024-09-16T02:53:27.871856+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:53:27.871856+0000","last_undegraded":"2024-09-16T02:53:27.871856+0000","last_fullsized":"2024-09-16T02:53:27.871856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":13471,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:14.263229+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:14.263229+0000","last_peered":"2024-09-16T02:54:14.263229+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:14.263229+0000","last_undegraded":"2024-09-16T02:54:14.263229+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":3121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3122},{"shards":"2,3","objects":902}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4053,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862494+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:53:27.862494+0000","last_peered":"2024-09-16T02:53:27.862494+0000","last_clean":"2024-09-16T02:53:27.862494+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:53:27.862494+0000","last_undegraded":"2024-09-16T02:53:27.862494+0000","last_fullsized":"2024-09-16T02:53:27.862494+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3995,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.862477+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:53:27.862477+0000","last_peered":"2024-09-16T02:53:27.862477+0000","last_clean":"2024-09-16T02:53:27.862477+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:53:27.862477+0000","last_undegraded":"2024-09-16T02:53:27.862477+0000","last_fullsized":"2024-09-16T02:53:27.862477+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1804,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30151,"num_bytes_recovered":252908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":404279296,"data_stored":819159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640565,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273372,"kb_used_data":125580,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94098468,"statfs":{"total":96636764160,"available":96356831232,"internally_reserved":0,"allocated":128593920,"data_stored":371288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673269,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":254912,"kb_used_data":125168,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94116928,"statfs":{"total":96636764160,"available":96375734272,"internally_reserved":0,"allocated":128172032,"data_stored":371185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986257,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128090112,"data_stored":253962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128700416,"data_stored":254111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:21.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:21.867 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:21.868 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:21.886 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":799,"stamp":"2024-09-16T02:54:21.780180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30839,"num_bytes_recovered":253596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":531400,"kb_used_data":253864,"kb_used_omap":3,"kb_used_meta":277500,"kb_avail":188212280,"statfs":{"total":193273528320,"available":192729374720,"internally_reserved":0,"allocated":259956736,"data_stored":769446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":284160906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2384,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":1192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002790"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4067,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923283+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:20.923283+0000","last_peered":"2024-09-16T02:54:20.923283+0000","last_clean":"2024-09-16T02:54:20.923283+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:20.923283+0000","last_undegraded":"2024-09-16T02:54:20.923283+0000","last_fullsized":"2024-09-16T02:54:20.923283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923222+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:20.923222+0000","last_peered":"2024-09-16T02:54:20.923222+0000","last_clean":"2024-09-16T02:54:20.923222+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:20.923222+0000","last_undegraded":"2024-09-16T02:54:20.923222+0000","last_fullsized":"2024-09-16T02:54:20.923222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4017,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923186+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:20.923186+0000","last_peered":"2024-09-16T02:54:20.923186+0000","last_clean":"2024-09-16T02:54:20.923186+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:20.923186+0000","last_undegraded":"2024-09-16T02:54:20.923186+0000","last_fullsized":"2024-09-16T02:54:20.923186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":15535,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:19.270943+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:19.270943+0000","last_peered":"2024-09-16T02:54:19.270943+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:19.270943+0000","last_undegraded":"2024-09-16T02:54:19.270943+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3811,"num_bytes_recovered":3809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3810},{"shards":"2,3","objects":214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923126+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:20.923126+0000","last_peered":"2024-09-16T02:54:20.923126+0000","last_clean":"2024-09-16T02:54:20.923126+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:20.923126+0000","last_undegraded":"2024-09-16T02:54:20.923126+0000","last_fullsized":"2024-09-16T02:54:20.923126+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3997,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923131+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:20.923131+0000","last_peered":"2024-09-16T02:54:20.923131+0000","last_clean":"2024-09-16T02:54:20.923131+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:20.923131+0000","last_undegraded":"2024-09-16T02:54:20.923131+0000","last_fullsized":"2024-09-16T02:54:20.923131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30839,"num_bytes_recovered":253596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407175168,"data_stored":846060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640565,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273372,"kb_used_data":125580,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94098468,"statfs":{"total":96636764160,"available":96356831232,"internally_reserved":0,"allocated":128593920,"data_stored":371288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258028,"kb_used_data":128284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113812,"statfs":{"total":96636764160,"available":96372543488,"internally_reserved":0,"allocated":131362816,"data_stored":398158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128700416,"data_stored":254111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:21.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:22.192 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:22.192 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:22.211 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2024-09-16T02:54:21.918500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30839,"num_bytes_recovered":253596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":531400,"kb_used_data":253864,"kb_used_omap":3,"kb_used_meta":277500,"kb_avail":188212280,"statfs":{"total":193273528320,"available":192729374720,"internally_reserved":0,"allocated":259956736,"data_stored":769446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":284160906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2384,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":1192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.140433"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4067,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923283+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:20.923283+0000","last_peered":"2024-09-16T02:54:20.923283+0000","last_clean":"2024-09-16T02:54:20.923283+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:20.923283+0000","last_undegraded":"2024-09-16T02:54:20.923283+0000","last_fullsized":"2024-09-16T02:54:20.923283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923222+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:20.923222+0000","last_peered":"2024-09-16T02:54:20.923222+0000","last_clean":"2024-09-16T02:54:20.923222+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:20.923222+0000","last_undegraded":"2024-09-16T02:54:20.923222+0000","last_fullsized":"2024-09-16T02:54:20.923222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4017,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923186+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:20.923186+0000","last_peered":"2024-09-16T02:54:20.923186+0000","last_clean":"2024-09-16T02:54:20.923186+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:20.923186+0000","last_undegraded":"2024-09-16T02:54:20.923186+0000","last_fullsized":"2024-09-16T02:54:20.923186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":15535,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:19.270943+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:19.270943+0000","last_peered":"2024-09-16T02:54:19.270943+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:19.270943+0000","last_undegraded":"2024-09-16T02:54:19.270943+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3811,"num_bytes_recovered":3809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3810},{"shards":"2,3","objects":214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923126+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:20.923126+0000","last_peered":"2024-09-16T02:54:20.923126+0000","last_clean":"2024-09-16T02:54:20.923126+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:20.923126+0000","last_undegraded":"2024-09-16T02:54:20.923126+0000","last_fullsized":"2024-09-16T02:54:20.923126+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3997,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923131+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:20.923131+0000","last_peered":"2024-09-16T02:54:20.923131+0000","last_clean":"2024-09-16T02:54:20.923131+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:20.923131+0000","last_undegraded":"2024-09-16T02:54:20.923131+0000","last_fullsized":"2024-09-16T02:54:20.923131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30839,"num_bytes_recovered":253596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407175168,"data_stored":846060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640565,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273372,"kb_used_data":125580,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94098468,"statfs":{"total":96636764160,"available":96356831232,"internally_reserved":0,"allocated":128593920,"data_stored":371288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258028,"kb_used_data":128284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113812,"statfs":{"total":96636764160,"available":96372543488,"internally_reserved":0,"allocated":131362816,"data_stored":398158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128700416,"data_stored":254111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:22.211 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-16T02:54:22.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "19'4024", 'reported_seq': 15535, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T02:54:19.270943+0000', 'last_change': '2024-09-16T02:53:26.115270+0000', 'last_active': '2024-09-16T02:54:19.270943+0000', 'last_peered': '2024-09-16T02:54:19.270943+0000', 'last_clean': '2024-09-16T02:46:36.933938+0000', 'last_became_active': '2024-09-16T02:47:26.530673+0000', 'last_became_peered': '2024-09-16T02:47:26.530673+0000', 'last_unstale': '2024-09-16T02:54:19.270943+0000', 'last_undegraded': '2024-09-16T02:54:19.270943+0000', 'last_fullsized': '2024-09-16T02:47:26.333875+0000', 'mapping_epoch': 28, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 19, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'last_clean_scrub_stamp': '2024-09-16T02:29:16.448904+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30223, 'num_objects': 4024, 'num_object_clones': 0, 'num_object_copies': 8048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 428, 'num_objects_unfound': 0, 'num_objects_dirty': 4024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4023, 'num_write_kb': 4047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3811, 'num_bytes_recovered': 3809, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3810}, {'shards': '2,3', 'objects': 214}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T02:54:22.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T02:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1522,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726454857,"num_in_osds":2,"osd_in_since":1726454799,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31923,"data_bytes":280863,"bytes_used":544153600,"bytes_avail":192729374720,"bytes_total":193273528320,"misplaced_objects":428,"misplaced_total":63846,"misplaced_ratio":0.0067036306111580993,"recovering_objects_per_sec":119,"recovering_bytes_per_sec":119,"recovering_keys_per_sec":0,"num_objects_recovered":1192,"num_bytes_recovered":1192,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T02:29:00:412138+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-16T02:51:25.730382+0000","services":{}},"progress_events":{}} 2024-09-16T02:54:22.634 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-16T02:54:22.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:22.943 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:22.943 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:22.961 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2024-09-16T02:54:21.918500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30839,"num_bytes_recovered":253596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":531400,"kb_used_data":253864,"kb_used_omap":3,"kb_used_meta":277500,"kb_avail":188212280,"statfs":{"total":193273528320,"available":192729374720,"internally_reserved":0,"allocated":259956736,"data_stored":769446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":284160906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2384,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1192,"num_bytes_recovered":1192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.140433"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4008,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.860453+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:53:27.860453+0000","last_peered":"2024-09-16T02:53:27.860453+0000","last_clean":"2024-09-16T02:53:27.860453+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:53:27.860453+0000","last_undegraded":"2024-09-16T02:53:27.860453+0000","last_fullsized":"2024-09-16T02:53:27.860453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4067,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923283+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:20.923283+0000","last_peered":"2024-09-16T02:54:20.923283+0000","last_clean":"2024-09-16T02:54:20.923283+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:20.923283+0000","last_undegraded":"2024-09-16T02:54:20.923283+0000","last_fullsized":"2024-09-16T02:54:20.923283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923222+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:20.923222+0000","last_peered":"2024-09-16T02:54:20.923222+0000","last_clean":"2024-09-16T02:54:20.923222+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:20.923222+0000","last_undegraded":"2024-09-16T02:54:20.923222+0000","last_fullsized":"2024-09-16T02:54:20.923222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4017,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923186+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:20.923186+0000","last_peered":"2024-09-16T02:54:20.923186+0000","last_clean":"2024-09-16T02:54:20.923186+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:20.923186+0000","last_undegraded":"2024-09-16T02:54:20.923186+0000","last_fullsized":"2024-09-16T02:54:20.923186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":15535,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T02:54:19.270943+0000","last_change":"2024-09-16T02:53:26.115270+0000","last_active":"2024-09-16T02:54:19.270943+0000","last_peered":"2024-09-16T02:54:19.270943+0000","last_clean":"2024-09-16T02:46:36.933938+0000","last_became_active":"2024-09-16T02:47:26.530673+0000","last_became_peered":"2024-09-16T02:47:26.530673+0000","last_unstale":"2024-09-16T02:54:19.270943+0000","last_undegraded":"2024-09-16T02:54:19.270943+0000","last_fullsized":"2024-09-16T02:47:26.333875+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3811,"num_bytes_recovered":3809,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3810},{"shards":"2,3","objects":214}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923126+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:20.923126+0000","last_peered":"2024-09-16T02:54:20.923126+0000","last_clean":"2024-09-16T02:54:20.923126+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:20.923126+0000","last_undegraded":"2024-09-16T02:54:20.923126+0000","last_fullsized":"2024-09-16T02:54:20.923126+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3997,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923131+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:20.923131+0000","last_peered":"2024-09-16T02:54:20.923131+0000","last_clean":"2024-09-16T02:54:20.923131+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:20.923131+0000","last_undegraded":"2024-09-16T02:54:20.923131+0000","last_fullsized":"2024-09-16T02:54:20.923131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4080,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:53:27.857401+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:53:27.857401+0000","last_peered":"2024-09-16T02:53:27.857401+0000","last_clean":"2024-09-16T02:53:27.857401+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:53:27.857401+0000","last_undegraded":"2024-09-16T02:53:27.857401+0000","last_fullsized":"2024-09-16T02:53:27.857401+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":428,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30839,"num_bytes_recovered":253596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407175168,"data_stored":846060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640565,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":273372,"kb_used_data":125580,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94098468,"statfs":{"total":96636764160,"available":96356831232,"internally_reserved":0,"allocated":128593920,"data_stored":371288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258028,"kb_used_data":128284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113812,"statfs":{"total":96636764160,"available":96372543488,"internally_reserved":0,"allocated":131362816,"data_stored":398158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128700416,"data_stored":254111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:25.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:26.271 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:26.271 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:26.289 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":802,"stamp":"2024-09-16T02:54:25.781304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":521632,"kb_used_data":256576,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188222048,"statfs":{"total":193273528320,"available":192739377152,"internally_reserved":0,"allocated":262733824,"data_stored":798971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1804,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":27102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002204"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4012,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.921681+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:54:21.921681+0000","last_peered":"2024-09-16T02:54:21.921681+0000","last_clean":"2024-09-16T02:54:21.921681+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:54:21.921681+0000","last_undegraded":"2024-09-16T02:54:21.921681+0000","last_fullsized":"2024-09-16T02:54:21.921681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4067,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923283+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:20.923283+0000","last_peered":"2024-09-16T02:54:20.923283+0000","last_clean":"2024-09-16T02:54:20.923283+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:20.923283+0000","last_undegraded":"2024-09-16T02:54:20.923283+0000","last_fullsized":"2024-09-16T02:54:20.923283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923222+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:20.923222+0000","last_peered":"2024-09-16T02:54:20.923222+0000","last_clean":"2024-09-16T02:54:20.923222+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:20.923222+0000","last_undegraded":"2024-09-16T02:54:20.923222+0000","last_fullsized":"2024-09-16T02:54:20.923222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4017,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923186+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:20.923186+0000","last_peered":"2024-09-16T02:54:20.923186+0000","last_clean":"2024-09-16T02:54:20.923186+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:20.923186+0000","last_undegraded":"2024-09-16T02:54:20.923186+0000","last_fullsized":"2024-09-16T02:54:20.923186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4072,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.927899+0000","last_change":"2024-09-16T02:54:21.927381+0000","last_active":"2024-09-16T02:54:21.927899+0000","last_peered":"2024-09-16T02:54:21.927899+0000","last_clean":"2024-09-16T02:54:21.927899+0000","last_became_active":"2024-09-16T02:54:21.924539+0000","last_became_peered":"2024-09-16T02:54:21.924539+0000","last_unstale":"2024-09-16T02:54:21.927899+0000","last_undegraded":"2024-09-16T02:54:21.927899+0000","last_fullsized":"2024-09-16T02:54:21.927899+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:36.455347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":30223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923126+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:20.923126+0000","last_peered":"2024-09-16T02:54:20.923126+0000","last_clean":"2024-09-16T02:54:20.923126+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:20.923126+0000","last_undegraded":"2024-09-16T02:54:20.923126+0000","last_fullsized":"2024-09-16T02:54:20.923126+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3997,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923131+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:20.923131+0000","last_peered":"2024-09-16T02:54:20.923131+0000","last_clean":"2024-09-16T02:54:20.923131+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:20.923131+0000","last_undegraded":"2024-09-16T02:54:20.923131+0000","last_fullsized":"2024-09-16T02:54:20.923131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4084,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.920368+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:54:21.920368+0000","last_peered":"2024-09-16T02:54:21.920368+0000","last_clean":"2024-09-16T02:54:21.920368+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:54:21.920368+0000","last_undegraded":"2024-09-16T02:54:21.920368+0000","last_fullsized":"2024-09-16T02:54:21.920368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405684224,"data_stored":871890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640566,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263604,"kb_used_data":128292,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94108236,"statfs":{"total":96636764160,"available":96366833664,"internally_reserved":0,"allocated":131371008,"data_stored":400813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258028,"kb_used_data":128284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113812,"statfs":{"total":96636764160,"available":96372543488,"internally_reserved":0,"allocated":131362816,"data_stored":398158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127209472,"data_stored":279941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:26.290 INFO:tasks.osd_recovery.ceph_manager:clean! 2024-09-16T02:54:26.290 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T02:54:26.306 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T02:54:26.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:26.611 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:26.628 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":802,"stamp":"2024-09-16T02:54:25.781304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":521632,"kb_used_data":256576,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188222048,"statfs":{"total":193273528320,"available":192739377152,"internally_reserved":0,"allocated":262733824,"data_stored":798971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1804,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":27102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002204"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4012,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.921681+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:54:21.921681+0000","last_peered":"2024-09-16T02:54:21.921681+0000","last_clean":"2024-09-16T02:54:21.921681+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:54:21.921681+0000","last_undegraded":"2024-09-16T02:54:21.921681+0000","last_fullsized":"2024-09-16T02:54:21.921681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4067,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923283+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:20.923283+0000","last_peered":"2024-09-16T02:54:20.923283+0000","last_clean":"2024-09-16T02:54:20.923283+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:20.923283+0000","last_undegraded":"2024-09-16T02:54:20.923283+0000","last_fullsized":"2024-09-16T02:54:20.923283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923222+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:20.923222+0000","last_peered":"2024-09-16T02:54:20.923222+0000","last_clean":"2024-09-16T02:54:20.923222+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:20.923222+0000","last_undegraded":"2024-09-16T02:54:20.923222+0000","last_fullsized":"2024-09-16T02:54:20.923222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4017,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923186+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:20.923186+0000","last_peered":"2024-09-16T02:54:20.923186+0000","last_clean":"2024-09-16T02:54:20.923186+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:20.923186+0000","last_undegraded":"2024-09-16T02:54:20.923186+0000","last_fullsized":"2024-09-16T02:54:20.923186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4072,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.927899+0000","last_change":"2024-09-16T02:54:21.927381+0000","last_active":"2024-09-16T02:54:21.927899+0000","last_peered":"2024-09-16T02:54:21.927899+0000","last_clean":"2024-09-16T02:54:21.927899+0000","last_became_active":"2024-09-16T02:54:21.924539+0000","last_became_peered":"2024-09-16T02:54:21.924539+0000","last_unstale":"2024-09-16T02:54:21.927899+0000","last_undegraded":"2024-09-16T02:54:21.927899+0000","last_fullsized":"2024-09-16T02:54:21.927899+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:36.455347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":30223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923126+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:20.923126+0000","last_peered":"2024-09-16T02:54:20.923126+0000","last_clean":"2024-09-16T02:54:20.923126+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:20.923126+0000","last_undegraded":"2024-09-16T02:54:20.923126+0000","last_fullsized":"2024-09-16T02:54:20.923126+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3997,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923131+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:20.923131+0000","last_peered":"2024-09-16T02:54:20.923131+0000","last_clean":"2024-09-16T02:54:20.923131+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:20.923131+0000","last_undegraded":"2024-09-16T02:54:20.923131+0000","last_fullsized":"2024-09-16T02:54:20.923131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4084,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.920368+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:54:21.920368+0000","last_peered":"2024-09-16T02:54:21.920368+0000","last_clean":"2024-09-16T02:54:21.920368+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:54:21.920368+0000","last_undegraded":"2024-09-16T02:54:21.920368+0000","last_fullsized":"2024-09-16T02:54:21.920368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405684224,"data_stored":871890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640566,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263604,"kb_used_data":128292,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94108236,"statfs":{"total":96636764160,"available":96366833664,"internally_reserved":0,"allocated":131371008,"data_stored":400813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258028,"kb_used_data":128284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113812,"statfs":{"total":96636764160,"available":96372543488,"internally_reserved":0,"allocated":131362816,"data_stored":398158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127209472,"data_stored":279941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:26.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:26.935 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:26.935 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:26.955 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":802,"stamp":"2024-09-16T02:54:25.781304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":521632,"kb_used_data":256576,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188222048,"statfs":{"total":193273528320,"available":192739377152,"internally_reserved":0,"allocated":262733824,"data_stored":798971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1804,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":27102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002204"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4012,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.921681+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:54:21.921681+0000","last_peered":"2024-09-16T02:54:21.921681+0000","last_clean":"2024-09-16T02:54:21.921681+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:54:21.921681+0000","last_undegraded":"2024-09-16T02:54:21.921681+0000","last_fullsized":"2024-09-16T02:54:21.921681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4067,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923283+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:20.923283+0000","last_peered":"2024-09-16T02:54:20.923283+0000","last_clean":"2024-09-16T02:54:20.923283+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:20.923283+0000","last_undegraded":"2024-09-16T02:54:20.923283+0000","last_fullsized":"2024-09-16T02:54:20.923283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923222+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:20.923222+0000","last_peered":"2024-09-16T02:54:20.923222+0000","last_clean":"2024-09-16T02:54:20.923222+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:20.923222+0000","last_undegraded":"2024-09-16T02:54:20.923222+0000","last_fullsized":"2024-09-16T02:54:20.923222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4017,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923186+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:20.923186+0000","last_peered":"2024-09-16T02:54:20.923186+0000","last_clean":"2024-09-16T02:54:20.923186+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:20.923186+0000","last_undegraded":"2024-09-16T02:54:20.923186+0000","last_fullsized":"2024-09-16T02:54:20.923186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4072,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.927899+0000","last_change":"2024-09-16T02:54:21.927381+0000","last_active":"2024-09-16T02:54:21.927899+0000","last_peered":"2024-09-16T02:54:21.927899+0000","last_clean":"2024-09-16T02:54:21.927899+0000","last_became_active":"2024-09-16T02:54:21.924539+0000","last_became_peered":"2024-09-16T02:54:21.924539+0000","last_unstale":"2024-09-16T02:54:21.927899+0000","last_undegraded":"2024-09-16T02:54:21.927899+0000","last_fullsized":"2024-09-16T02:54:21.927899+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:36.455347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":30223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923126+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:20.923126+0000","last_peered":"2024-09-16T02:54:20.923126+0000","last_clean":"2024-09-16T02:54:20.923126+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:20.923126+0000","last_undegraded":"2024-09-16T02:54:20.923126+0000","last_fullsized":"2024-09-16T02:54:20.923126+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3997,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923131+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:20.923131+0000","last_peered":"2024-09-16T02:54:20.923131+0000","last_clean":"2024-09-16T02:54:20.923131+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:20.923131+0000","last_undegraded":"2024-09-16T02:54:20.923131+0000","last_fullsized":"2024-09-16T02:54:20.923131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4084,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.920368+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:54:21.920368+0000","last_peered":"2024-09-16T02:54:21.920368+0000","last_clean":"2024-09-16T02:54:21.920368+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:54:21.920368+0000","last_undegraded":"2024-09-16T02:54:21.920368+0000","last_fullsized":"2024-09-16T02:54:21.920368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405684224,"data_stored":871890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640566,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263604,"kb_used_data":128292,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94108236,"statfs":{"total":96636764160,"available":96366833664,"internally_reserved":0,"allocated":131371008,"data_stored":400813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258028,"kb_used_data":128284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113812,"statfs":{"total":96636764160,"available":96372543488,"internally_reserved":0,"allocated":131362816,"data_stored":398158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127209472,"data_stored":279941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:26.956 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T02:54:26.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:27.259 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:27.260 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:27.277 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":802,"stamp":"2024-09-16T02:54:25.781304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":521632,"kb_used_data":256576,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188222048,"statfs":{"total":193273528320,"available":192739377152,"internally_reserved":0,"allocated":262733824,"data_stored":798971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1804,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":27102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002204"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4012,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.921681+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:54:21.921681+0000","last_peered":"2024-09-16T02:54:21.921681+0000","last_clean":"2024-09-16T02:54:21.921681+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:54:21.921681+0000","last_undegraded":"2024-09-16T02:54:21.921681+0000","last_fullsized":"2024-09-16T02:54:21.921681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4067,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923283+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:20.923283+0000","last_peered":"2024-09-16T02:54:20.923283+0000","last_clean":"2024-09-16T02:54:20.923283+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:20.923283+0000","last_undegraded":"2024-09-16T02:54:20.923283+0000","last_fullsized":"2024-09-16T02:54:20.923283+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4126,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923222+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:20.923222+0000","last_peered":"2024-09-16T02:54:20.923222+0000","last_clean":"2024-09-16T02:54:20.923222+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:20.923222+0000","last_undegraded":"2024-09-16T02:54:20.923222+0000","last_fullsized":"2024-09-16T02:54:20.923222+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4017,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923186+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:20.923186+0000","last_peered":"2024-09-16T02:54:20.923186+0000","last_clean":"2024-09-16T02:54:20.923186+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:20.923186+0000","last_undegraded":"2024-09-16T02:54:20.923186+0000","last_fullsized":"2024-09-16T02:54:20.923186+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4072,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.927899+0000","last_change":"2024-09-16T02:54:21.927381+0000","last_active":"2024-09-16T02:54:21.927899+0000","last_peered":"2024-09-16T02:54:21.927899+0000","last_clean":"2024-09-16T02:54:21.927899+0000","last_became_active":"2024-09-16T02:54:21.924539+0000","last_became_peered":"2024-09-16T02:54:21.924539+0000","last_unstale":"2024-09-16T02:54:21.927899+0000","last_undegraded":"2024-09-16T02:54:21.927899+0000","last_fullsized":"2024-09-16T02:54:21.927899+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:36.455347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":30223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923126+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:20.923126+0000","last_peered":"2024-09-16T02:54:20.923126+0000","last_clean":"2024-09-16T02:54:20.923126+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:20.923126+0000","last_undegraded":"2024-09-16T02:54:20.923126+0000","last_fullsized":"2024-09-16T02:54:20.923126+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3997,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T02:54:20.923131+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:20.923131+0000","last_peered":"2024-09-16T02:54:20.923131+0000","last_clean":"2024-09-16T02:54:20.923131+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:20.923131+0000","last_undegraded":"2024-09-16T02:54:20.923131+0000","last_fullsized":"2024-09-16T02:54:20.923131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4084,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.920368+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:54:21.920368+0000","last_peered":"2024-09-16T02:54:21.920368+0000","last_clean":"2024-09-16T02:54:21.920368+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:54:21.920368+0000","last_undegraded":"2024-09-16T02:54:21.920368+0000","last_fullsized":"2024-09-16T02:54:21.920368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405684224,"data_stored":871890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640566,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263604,"kb_used_data":128292,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94108236,"statfs":{"total":96636764160,"available":96366833664,"internally_reserved":0,"allocated":131371008,"data_stored":400813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673270,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258028,"kb_used_data":128284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113812,"statfs":{"total":96636764160,"available":96372543488,"internally_reserved":0,"allocated":131362816,"data_stored":398158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127209472,"data_stored":279941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:27.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:54:27.573 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:27.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":47,"fsid":"eabc9450-e776-4be8-8adb-00ce348fa2c2","created":"2024-09-16T02:29:00.413885+0000","modified":"2024-09-16T02:54:21.912749+0000","last_up_change":"2024-09-16T02:47:37.699678+0000","last_in_change":"2024-09-16T02:46:39.798102+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T02:29:16.348818+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":0.5,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12c8d8a5-d99d-4bfc-99dd-7b19e351ed44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":683366439}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":683366439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":683366439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":683366439}]},"public_addr":"172.21.15.137:6804/683366439","cluster_addr":"172.21.15.137:6805/683366439","heartbeat_back_addr":"172.21.15.137:6807/683366439","heartbeat_front_addr":"172.21.15.137:6806/683366439","state":["exists","up"]},{"osd":1,"uuid":"8a607138-b9f7-49d3-941d-ca065ebe247f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":4172164026}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":4172164026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":4172164026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":4172164026}]},"public_addr":"172.21.15.137:6812/4172164026","cluster_addr":"172.21.15.137:6813/4172164026","heartbeat_back_addr":"172.21.15.137:6815/4172164026","heartbeat_front_addr":"172.21.15.137:6814/4172164026","state":["exists","up"]},{"osd":2,"uuid":"6fb52c0c-2fc1-4845-a3ba-09f9da4a6a82","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":23,"up_from":27,"up_thru":45,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6800","nonce":4262355089}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":4262355089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":4262355089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":4262355089}]},"public_addr":"172.21.15.137:6800/4262355089","cluster_addr":"172.21.15.137:6801/4262355089","heartbeat_back_addr":"172.21.15.137:6803/4262355089","heartbeat_front_addr":"172.21.15.137:6802/4262355089","state":["exists","up"]},{"osd":3,"uuid":"54f6e8af-e613-4fe3-a2ae-f1ddeed4f78a","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":23,"up_from":31,"up_thru":18,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":1833792554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1833792554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1833792554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1833792554}]},"public_addr":"172.21.15.137:6808/1833792554","cluster_addr":"172.21.15.137:6809/1833792554","heartbeat_back_addr":"172.21.15.137:6811/1833792554","heartbeat_front_addr":"172.21.15.137:6810/1833792554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.492963+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:29:08.693906+0000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-16T02:47:12.737384+0000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T02:29:08.471476+0000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-16T02:47:12.737384+0000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T02:29:08.155330+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:54:28.593 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T02:54:28.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:54:28.754 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:54:28.754 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:54:28.754 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:54:28.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T02:54:29.056 INFO:teuthology.orchestra.run.smithi137.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T02:54:29.076 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T02:54:29.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:54:29.213 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:54:29.213 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:54:29.213 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:54:29.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T02:54:29.512 INFO:teuthology.orchestra.run.smithi137.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T02:54:29.529 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T02:54:29.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:54:29.675 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:54:29.676 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:54:29.676 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:54:29.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T02:54:30.081 INFO:teuthology.orchestra.run.smithi137.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T02:54:30.106 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T02:54:30.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:54:30.291 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-16T02:54:30.291 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:54:30.291 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-16T02:54:30.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T02:54:30.753 INFO:teuthology.orchestra.run.smithi137.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T02:54:30.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:31.309 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:31.309 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:31.338 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":804,"stamp":"2024-09-16T02:54:29.782277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":521656,"kb_used_data":256600,"kb_used_omap":3,"kb_used_meta":265020,"kb_avail":188222024,"statfs":{"total":193273528320,"available":192739352576,"internally_reserved":0,"allocated":262758400,"data_stored":806894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":271381386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-428,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":26414,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.868465"},"pg_stats":[{"pgid":"1.7","version":"19'3954","reported_seq":4012,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.921681+0000","last_change":"2024-09-16T02:49:46.804776+0000","last_active":"2024-09-16T02:54:21.921681+0000","last_peered":"2024-09-16T02:54:21.921681+0000","last_clean":"2024-09-16T02:54:21.921681+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:54:21.921681+0000","last_undegraded":"2024-09-16T02:54:21.921681+0000","last_fullsized":"2024-09-16T02:54:21.921681+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3954,"log_dups_size":0,"ondisk_log_size":3954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:01:56.270277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"19'3985","reported_seq":4069,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.923488+0000","last_change":"2024-09-16T02:48:54.774277+0000","last_active":"2024-09-16T02:54:21.923488+0000","last_peered":"2024-09-16T02:54:21.923488+0000","last_clean":"2024-09-16T02:54:21.923488+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:21.923488+0000","last_undegraded":"2024-09-16T02:54:21.923488+0000","last_fullsized":"2024-09-16T02:54:21.923488+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:14:57.515019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"19'4060","reported_seq":4128,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.923406+0000","last_change":"2024-09-16T02:51:36.389793+0000","last_active":"2024-09-16T02:54:21.923406+0000","last_peered":"2024-09-16T02:54:21.923406+0000","last_clean":"2024-09-16T02:54:21.923406+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:21.923406+0000","last_undegraded":"2024-09-16T02:54:21.923406+0000","last_fullsized":"2024-09-16T02:54:21.923406+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4060,"log_dups_size":0,"ondisk_log_size":4060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.675603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'3956","reported_seq":4019,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.923326+0000","last_change":"2024-09-16T02:53:27.869800+0000","last_active":"2024-09-16T02:54:21.923326+0000","last_peered":"2024-09-16T02:54:21.923326+0000","last_clean":"2024-09-16T02:54:21.923326+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:21.923326+0000","last_undegraded":"2024-09-16T02:54:21.923326+0000","last_fullsized":"2024-09-16T02:54:21.923326+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:16.511958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'4024","reported_seq":4072,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.927899+0000","last_change":"2024-09-16T02:54:21.927381+0000","last_active":"2024-09-16T02:54:21.927899+0000","last_peered":"2024-09-16T02:54:21.927899+0000","last_clean":"2024-09-16T02:54:21.927899+0000","last_became_active":"2024-09-16T02:54:21.924539+0000","last_became_peered":"2024-09-16T02:54:21.924539+0000","last_unstale":"2024-09-16T02:54:21.927899+0000","last_undegraded":"2024-09-16T02:54:21.927899+0000","last_fullsized":"2024-09-16T02:54:21.927899+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:36.455347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":30223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"19'3998","reported_seq":4057,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.923315+0000","last_change":"2024-09-16T02:50:40.848495+0000","last_active":"2024-09-16T02:54:21.923315+0000","last_peered":"2024-09-16T02:54:21.923315+0000","last_clean":"2024-09-16T02:54:21.923315+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:21.923315+0000","last_undegraded":"2024-09-16T02:54:21.923315+0000","last_fullsized":"2024-09-16T02:54:21.923315+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:02.436308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3910","reported_seq":3999,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.923299+0000","last_change":"2024-09-16T02:48:00.806411+0000","last_active":"2024-09-16T02:54:21.923299+0000","last_peered":"2024-09-16T02:54:21.923299+0000","last_clean":"2024-09-16T02:54:21.923299+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:21.923299+0000","last_undegraded":"2024-09-16T02:54:21.923299+0000","last_fullsized":"2024-09-16T02:54:21.923299+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:49.935107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'4037","reported_seq":4084,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:21.920368+0000","last_change":"2024-09-16T02:52:34.021078+0000","last_active":"2024-09-16T02:54:21.920368+0000","last_peered":"2024-09-16T02:54:21.920368+0000","last_clean":"2024-09-16T02:54:21.920368+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:54:21.920368+0000","last_undegraded":"2024-09-16T02:54:21.920368+0000","last_fullsized":"2024-09-16T02:54:21.920368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.950437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":400744448,"data_stored":844490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31924,"ondisk_log_size":31924,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640567,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":263612,"kb_used_data":128300,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94108228,"statfs":{"total":96636764160,"available":96366825472,"internally_reserved":0,"allocated":131379200,"data_stored":403447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673271,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":258044,"kb_used_data":128300,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":94113796,"statfs":{"total":96636764160,"available":96372527104,"internally_reserved":0,"allocated":131379200,"data_stored":403447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986259,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122269696,"data_stored":252541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:31.342 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.342 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.343 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.343 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.343 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.343 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.343 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.343 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:31.343 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T02:54:51.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:54:56.310 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:54:56.310 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:54:56.516 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":817,"stamp":"2024-09-16T02:54:55.789310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59216,"ondisk_log_size":59216,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":585528,"kb_used_data":256600,"kb_used_omap":3,"kb_used_meta":328892,"kb_avail":188158152,"statfs":{"total":193273528320,"available":192673947648,"internally_reserved":0,"allocated":262758400,"data_stored":806894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":336786314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,12,10,25,25,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004725"},"pg_stats":[{"pgid":"1.7","version":"47'7328","reported_seq":7616,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:53.124411+0000","last_change":"2024-09-16T02:54:31.830698+0000","last_active":"2024-09-16T02:54:53.124411+0000","last_peered":"2024-09-16T02:54:53.124411+0000","last_clean":"2024-09-16T02:54:53.124411+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:54:53.124411+0000","last_undegraded":"2024-09-16T02:54:53.124411+0000","last_fullsized":"2024-09-16T02:54:53.124411+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3374,"log_size":7328,"log_dups_size":0,"ondisk_log_size":7328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 21s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'7464","reported_seq":7786,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:52.732946+0000","last_change":"2024-09-16T02:54:30.660254+0000","last_active":"2024-09-16T02:54:52.732946+0000","last_peered":"2024-09-16T02:54:52.732946+0000","last_clean":"2024-09-16T02:54:52.732946+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:52.732946+0000","last_undegraded":"2024-09-16T02:54:52.732946+0000","last_fullsized":"2024-09-16T02:54:52.732946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3494,"log_size":7464,"log_dups_size":0,"ondisk_log_size":7464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 22s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"47'7884","reported_seq":8198,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:52.720550+0000","last_change":"2024-09-16T02:54:29.650710+0000","last_active":"2024-09-16T02:54:52.720550+0000","last_peered":"2024-09-16T02:54:52.720550+0000","last_clean":"2024-09-16T02:54:52.720550+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:52.720550+0000","last_undegraded":"2024-09-16T02:54:52.720550+0000","last_fullsized":"2024-09-16T02:54:52.720550+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3824,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 23s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"47'7240","reported_seq":7522,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:52.732225+0000","last_change":"2024-09-16T02:54:31.645508+0000","last_active":"2024-09-16T02:54:52.732225+0000","last_peered":"2024-09-16T02:54:52.732225+0000","last_clean":"2024-09-16T02:54:52.732225+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:52.732225+0000","last_undegraded":"2024-09-16T02:54:52.732225+0000","last_fullsized":"2024-09-16T02:54:52.732225+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3284,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 21s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"47'7488","reported_seq":7757,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:53.060219+0000","last_change":"2024-09-16T02:54:30.829417+0000","last_active":"2024-09-16T02:54:53.060219+0000","last_peered":"2024-09-16T02:54:53.060219+0000","last_clean":"2024-09-16T02:54:53.060219+0000","last_became_active":"2024-09-16T02:54:21.924539+0000","last_became_peered":"2024-09-16T02:54:21.924539+0000","last_unstale":"2024-09-16T02:54:53.060219+0000","last_undegraded":"2024-09-16T02:54:53.060219+0000","last_fullsized":"2024-09-16T02:54:53.060219+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3464,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 22s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":30223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'7117","reported_seq":7389,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:52.706384+0000","last_change":"2024-09-16T02:54:32.653126+0000","last_active":"2024-09-16T02:54:52.706384+0000","last_peered":"2024-09-16T02:54:52.706384+0000","last_clean":"2024-09-16T02:54:52.706384+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:52.706384+0000","last_undegraded":"2024-09-16T02:54:52.706384+0000","last_fullsized":"2024-09-16T02:54:52.706384+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3119,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 20s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"47'6924","reported_seq":7214,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:52.725641+0000","last_change":"2024-09-16T02:54:33.688753+0000","last_active":"2024-09-16T02:54:52.725641+0000","last_peered":"2024-09-16T02:54:52.725641+0000","last_clean":"2024-09-16T02:54:52.725641+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:52.725641+0000","last_undegraded":"2024-09-16T02:54:52.725641+0000","last_fullsized":"2024-09-16T02:54:52.725641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3014,"log_size":6924,"log_dups_size":0,"ondisk_log_size":6924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 19s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"47'7771","reported_seq":8073,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T02:54:53.031579+0000","last_change":"2024-09-16T02:54:29.868850+0000","last_active":"2024-09-16T02:54:53.031579+0000","last_peered":"2024-09-16T02:54:53.031579+0000","last_clean":"2024-09-16T02:54:53.031579+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:54:53.031579+0000","last_undegraded":"2024-09-16T02:54:53.031579+0000","last_fullsized":"2024-09-16T02:54:53.031579+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:29:16.448904+0000","last_clean_scrub_stamp":"2024-09-16T02:29:16.448904+0000","objects_scrubbed":3749,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 23s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376135680,"data_stored":799191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59216,"ondisk_log_size":59216,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640572,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299708,"kb_used_data":128300,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":94072132,"statfs":{"total":96636764160,"available":96329863168,"internally_reserved":0,"allocated":131379200,"data_stored":403447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,7,4,9,17],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673276,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":285820,"kb_used_data":128300,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":94086020,"statfs":{"total":96636764160,"available":96344084480,"internally_reserved":0,"allocated":131379200,"data_stored":403447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,6,16,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986264,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97660928,"data_stored":207242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:54:56.517 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.517 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.517 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.518 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.518 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.518 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.518 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.518 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T02:29:16.448904+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=29, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=54, tm_sec=27, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:54:56.518 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T02:55:16.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:55:17.026 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-16T02:55:17.026 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-16T02:55:17.044 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":827,"stamp":"2024-09-16T02:55:15.795741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63847,"ondisk_log_size":63847,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":593848,"kb_used_data":256600,"kb_used_omap":3,"kb_used_meta":337212,"kb_avail":188149832,"statfs":{"total":193273528320,"available":192665427968,"internally_reserved":0,"allocated":262758400,"data_stored":806894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3190,"internal_metadata":345305994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002928"},"pg_stats":[{"pgid":"1.7","version":"47'7908","reported_seq":8239,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:57.460485+0000","last_change":"2024-09-16T02:54:57.460485+0000","last_active":"2024-09-16T02:54:57.460485+0000","last_peered":"2024-09-16T02:54:57.460485+0000","last_clean":"2024-09-16T02:54:57.460485+0000","last_became_active":"2024-09-16T02:49:46.800705+0000","last_became_peered":"2024-09-16T02:49:46.800705+0000","last_unstale":"2024-09-16T02:54:57.460485+0000","last_undegraded":"2024-09-16T02:54:57.460485+0000","last_fullsized":"2024-09-16T02:54:57.460485+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'7908","last_scrub_stamp":"2024-09-16T02:54:57.460368+0000","last_deep_scrub":"47'7908","last_deep_scrub_stamp":"2024-09-16T02:54:57.460368+0000","last_clean_scrub_stamp":"2024-09-16T02:54:57.460368+0000","objects_scrubbed":3954,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:39:50.032009+0000","scrub_duration":25631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3954,"num_objects":3954,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3954,"num_write_kb":3954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3954,"num_bytes_recovered":3954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'7970","reported_seq":8329,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:56.753146+0000","last_change":"2024-09-16T02:54:56.753146+0000","last_active":"2024-09-16T02:54:56.753146+0000","last_peered":"2024-09-16T02:54:56.753146+0000","last_clean":"2024-09-16T02:54:56.753146+0000","last_became_active":"2024-09-16T02:48:54.767171+0000","last_became_peered":"2024-09-16T02:48:54.767171+0000","last_unstale":"2024-09-16T02:54:56.753146+0000","last_undegraded":"2024-09-16T02:54:56.753146+0000","last_fullsized":"2024-09-16T02:54:56.753146+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'7970","last_scrub_stamp":"2024-09-16T02:54:56.753022+0000","last_deep_scrub":"47'7970","last_deep_scrub_stamp":"2024-09-16T02:54:56.753022+0000","last_clean_scrub_stamp":"2024-09-16T02:54:56.753022+0000","objects_scrubbed":3985,"log_size":7970,"log_dups_size":0,"ondisk_log_size":7970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":27,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:32:59.116337+0000","scrub_duration":26091,"objects_trimmed":0,"snaptrim_duration":0.0014837629999999999,"stat_sum":{"num_bytes":56385,"num_objects":3985,"num_object_clones":0,"num_object_copies":7970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":4033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":56385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"47'8120","reported_seq":8468,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:54.987801+0000","last_change":"2024-09-16T02:54:54.987801+0000","last_active":"2024-09-16T02:54:54.987801+0000","last_peered":"2024-09-16T02:54:54.987801+0000","last_clean":"2024-09-16T02:54:54.987801+0000","last_became_active":"2024-09-16T02:51:36.383610+0000","last_became_peered":"2024-09-16T02:51:36.383610+0000","last_unstale":"2024-09-16T02:54:54.987801+0000","last_undegraded":"2024-09-16T02:54:54.987801+0000","last_fullsized":"2024-09-16T02:54:54.987801+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8120","last_scrub_stamp":"2024-09-16T02:54:54.987684+0000","last_deep_scrub":"47'8120","last_deep_scrub_stamp":"2024-09-16T02:54:54.987684+0000","last_clean_scrub_stamp":"2024-09-16T02:54:54.987684+0000","objects_scrubbed":4060,"log_size":8120,"log_dups_size":0,"ondisk_log_size":8120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:35:27.120758+0000","scrub_duration":25339,"objects_trimmed":0,"snaptrim_duration":0.00093271300000000005,"stat_sum":{"num_bytes":30283,"num_objects":4060,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4060,"num_write_kb":4084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4060,"num_bytes_recovered":30283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"47'7912","reported_seq":8248,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:57.609989+0000","last_change":"2024-09-16T02:54:57.609989+0000","last_active":"2024-09-16T02:54:57.609989+0000","last_peered":"2024-09-16T02:54:57.609989+0000","last_clean":"2024-09-16T02:54:57.609989+0000","last_became_active":"2024-09-16T02:53:27.867125+0000","last_became_peered":"2024-09-16T02:53:27.867125+0000","last_unstale":"2024-09-16T02:54:57.609989+0000","last_undegraded":"2024-09-16T02:54:57.609989+0000","last_fullsized":"2024-09-16T02:54:57.609989+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'7912","last_scrub_stamp":"2024-09-16T02:54:57.609860+0000","last_deep_scrub":"47'7912","last_deep_scrub_stamp":"2024-09-16T02:54:57.609860+0000","last_clean_scrub_stamp":"2024-09-16T02:54:57.609860+0000","objects_scrubbed":3956,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:07.249171+0000","scrub_duration":25963,"objects_trimmed":0,"snaptrim_duration":0.001597977,"stat_sum":{"num_bytes":56356,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":56356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"47'8048","reported_seq":8374,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:57.496521+0000","last_change":"2024-09-16T02:54:57.496521+0000","last_active":"2024-09-16T02:54:57.496521+0000","last_peered":"2024-09-16T02:54:57.496521+0000","last_clean":"2024-09-16T02:54:57.496521+0000","last_became_active":"2024-09-16T02:54:21.924539+0000","last_became_peered":"2024-09-16T02:54:21.924539+0000","last_unstale":"2024-09-16T02:54:57.496521+0000","last_undegraded":"2024-09-16T02:54:57.496521+0000","last_fullsized":"2024-09-16T02:54:57.496521+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8048","last_scrub_stamp":"2024-09-16T02:54:57.496416+0000","last_deep_scrub":"47'8048","last_deep_scrub_stamp":"2024-09-16T02:54:57.496416+0000","last_clean_scrub_stamp":"2024-09-16T02:54:57.496416+0000","objects_scrubbed":4024,"log_size":8048,"log_dups_size":0,"ondisk_log_size":8048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":27,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:16.704371+0000","scrub_duration":26667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30223,"num_objects":4024,"num_object_clones":0,"num_object_copies":8048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":4047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":30223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'7995","reported_seq":8330,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:58.225377+0000","last_change":"2024-09-16T02:54:58.225377+0000","last_active":"2024-09-16T02:54:58.225377+0000","last_peered":"2024-09-16T02:54:58.225377+0000","last_clean":"2024-09-16T02:54:58.225377+0000","last_became_active":"2024-09-16T02:50:40.844553+0000","last_became_peered":"2024-09-16T02:50:40.844553+0000","last_unstale":"2024-09-16T02:54:58.225377+0000","last_undegraded":"2024-09-16T02:54:58.225377+0000","last_fullsized":"2024-09-16T02:54:58.225377+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'7995","last_scrub_stamp":"2024-09-16T02:54:58.225296+0000","last_deep_scrub":"47'7995","last_deep_scrub_stamp":"2024-09-16T02:54:58.225296+0000","last_clean_scrub_stamp":"2024-09-16T02:54:58.225296+0000","objects_scrubbed":3997,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:58.115141+0000","scrub_duration":25571,"objects_trimmed":0,"snaptrim_duration":0.0048370160000000004,"stat_sum":{"num_bytes":30215,"num_objects":3997,"num_object_clones":0,"num_object_copies":7994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":4022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3998,"num_bytes_recovered":30234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"47'7820","reported_seq":8179,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:58.269840+0000","last_change":"2024-09-16T02:54:58.269840+0000","last_active":"2024-09-16T02:54:58.269840+0000","last_peered":"2024-09-16T02:54:58.269840+0000","last_clean":"2024-09-16T02:54:58.269840+0000","last_became_active":"2024-09-16T02:48:00.801339+0000","last_became_peered":"2024-09-16T02:48:00.801339+0000","last_unstale":"2024-09-16T02:54:58.269840+0000","last_undegraded":"2024-09-16T02:54:58.269840+0000","last_fullsized":"2024-09-16T02:54:58.269840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'7820","last_scrub_stamp":"2024-09-16T02:54:58.269760+0000","last_deep_scrub":"47'7820","last_deep_scrub_stamp":"2024-09-16T02:54:58.269760+0000","last_clean_scrub_stamp":"2024-09-16T02:54:58.269760+0000","objects_scrubbed":3910,"log_size":7820,"log_dups_size":0,"ondisk_log_size":7820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":25,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:33.076389+0000","scrub_duration":24579,"objects_trimmed":0,"snaptrim_duration":0.0017117829999999999,"stat_sum":{"num_bytes":17010,"num_objects":3910,"num_object_clones":0,"num_object_copies":7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":3922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3038,"num_bytes_recovered":16138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"47'8074","reported_seq":8400,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T02:54:55.635805+0000","last_change":"2024-09-16T02:54:55.635805+0000","last_active":"2024-09-16T02:54:55.635805+0000","last_peered":"2024-09-16T02:54:55.635805+0000","last_clean":"2024-09-16T02:54:55.635805+0000","last_became_active":"2024-09-16T02:52:34.018195+0000","last_became_peered":"2024-09-16T02:52:34.018195+0000","last_unstale":"2024-09-16T02:54:55.635805+0000","last_undegraded":"2024-09-16T02:54:55.635805+0000","last_fullsized":"2024-09-16T02:54:55.635805+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8074","last_scrub_stamp":"2024-09-16T02:54:55.635716+0000","last_deep_scrub":"47'8074","last_deep_scrub_stamp":"2024-09-16T02:54:55.635716+0000","last_clean_scrub_stamp":"2024-09-16T02:54:55.635716+0000","objects_scrubbed":4037,"log_size":8074,"log_dups_size":0,"ondisk_log_size":8074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:10.759798+0000","scrub_duration":25767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56437,"num_objects":4037,"num_object_clones":0,"num_object_copies":8074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4037,"num_bytes_recovered":56437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":280863,"num_objects":31923,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31923,"num_write_kb":32151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31053,"num_bytes_recovered":280010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356438016,"data_stored":755091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63847,"ondisk_log_size":63847,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":2,"up_from":0,"seq":115964117088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640576,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312188,"kb_used_data":128300,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":94059652,"statfs":{"total":96636764160,"available":96317083648,"internally_reserved":0,"allocated":131379200,"data_stored":403447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673280,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":281660,"kb_used_data":128300,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94090180,"statfs":{"total":96636764160,"available":96348344320,"internally_reserved":0,"allocated":131379200,"data_stored":403447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":133143986268,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":130985984,"data_stored":280863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77963264,"data_stored":163142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16502784,"data_stored":30223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:55:17.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T02:55:17.368 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T02:55:17.369 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T02:55:17.369 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T02:55:17.369 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:17.470 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T02:55:17.470 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T02:55:17.470 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:17.563 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T02:55:17.563 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T02:55:17.563 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:21.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-16T02:55:23.664 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T02:55:23.665 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T02:55:23.665 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:23.729 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T02:55:23.749 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T02:55:23.749 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T02:55:23.749 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:23.779 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T02:55:23.780 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T02:55:23.780 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T02:55:23.780 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:23.842 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T02:55:23.842 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T02:55:23.842 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:23.902 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T02:55:23.902 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T02:55:23.903 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:55:23.961 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T02:55:23.961 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T02:55:23.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-16T02:55:24.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:55:24.030 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T02:55:24.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:55:24.394 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T02:55:24.660 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:55:24.660 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T02:55:24.837 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:55:24.838 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T02:55:25.034 INFO:tasks.ceph:Archiving mon data... 2024-09-16T02:55:25.036 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763/data/mon.a.tgz 2024-09-16T02:55:25.036 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-16T02:55:25.050 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.hdOYt7Glm1 2024-09-16T02:55:25.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.hdOYt7Glm1 2024-09-16T02:55:25.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.hdOYt7Glm1 2024-09-16T02:55:25.250 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.hdOYt7Glm1 is 269KB 2024-09-16T02:55:25.311 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.hdOYt7Glm1 2024-09-16T02:55:25.318 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763/data/mon.b.tgz 2024-09-16T02:55:25.318 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-16T02:55:25.366 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.2rkNp2E1aQ 2024-09-16T02:55:25.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.2rkNp2E1aQ 2024-09-16T02:55:25.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.2rkNp2E1aQ 2024-09-16T02:55:25.538 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.2rkNp2E1aQ is 270KB 2024-09-16T02:55:25.595 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.2rkNp2E1aQ 2024-09-16T02:55:25.601 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763/data/mon.c.tgz 2024-09-16T02:55:25.601 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-16T02:55:25.650 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.yU2iH3LV2u 2024-09-16T02:55:25.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yU2iH3LV2u 2024-09-16T02:55:25.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.yU2iH3LV2u 2024-09-16T02:55:25.813 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.yU2iH3LV2u is 270KB 2024-09-16T02:55:25.875 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.yU2iH3LV2u 2024-09-16T02:55:25.881 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T02:55:25.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T02:55:26.012 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T02:55:26.012 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T02:55:26.014 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763/remote/smithi137/crash 2024-09-16T02:55:26.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T02:55:26.066 INFO:tasks.ceph:Compressing logs... 2024-09-16T02:55:26.066 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T02:55:26.130 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51485.log 2024-09-16T02:55:26.130 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45243.log 2024-09-16T02:55:26.131 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43887.log 2024-09-16T02:55:26.131 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61190.log 2024-09-16T02:55:26.132 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66913.log 2024-09-16T02:55:26.133 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2024-09-16T02:55:26.133 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45503.log 2024-09-16T02:55:26.133 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34128.log 2024-09-16T02:55:26.133 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35974.log 2024-09-16T02:55:26.134 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69383.log 2024-09-16T02:55:26.134 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56978.log 2024-09-16T02:55:26.134 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65535.log 2024-09-16T02:55:26.135 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45633.log 2024-09-16T02:55:26.135 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50939.log 2024-09-16T02:55:26.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-16T02:55:26.136 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52474.log 2024-09-16T02:55:26.136 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51043.log 2024-09-16T02:55:26.137 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66653.log 2024-09-16T02:55:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54398.log 2024-09-16T02:55:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67667.log 2024-09-16T02:55:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32037.log 2024-09-16T02:55:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49431.log 2024-09-16T02:55:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64546.log 2024-09-16T02:55:26.138 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58060.log 2024-09-16T02:55:26.139 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50523.log 2024-09-16T02:55:26.139 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51694.log 2024-09-16T02:55:26.140 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56406.log 2024-09-16T02:55:26.140 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40057.log 2024-09-16T02:55:26.344 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34180.log 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45243.log: /var/log/ceph/ceph-client.admin.40057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51485.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49717.log 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61190.log: /var/log/ceph/ceph-client.admin.66913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40057.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45243.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61190.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66913.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34180.log: /var/log/ceph/ceph-client.admin.34128.log: /var/log/ceph/ceph-client.admin.43887.log: /var/log/ceph/ceph-client.admin.35974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45503.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45425.log 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34180.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34128.log.gz 2024-09-16T02:55:26.345 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43887.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35974.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69383.log: /var/log/ceph/ceph-client.admin.49717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69383.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65535.log: /var/log/ceph/ceph-client.admin.56978.log: /var/log/ceph/ceph-client.admin.45633.log: /var/log/ceph/ceph-client.admin.50939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49717.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52838.log 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56978.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65535.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45633.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50939.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52474.log: /var/log/ceph/ceph-client.admin.51043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52474.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51043.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48443.log 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45425.log.gz 2024-09-16T02:55:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66653.log: /var/log/ceph/ceph-client.admin.54398.log: /var/log/ceph/ceph-client.admin.67667.log: /var/log/ceph/ceph-client.admin.32037.log: /var/log/ceph/ceph-client.admin.52838.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66653.log.gz -- replaced with /var/log/ceph/ceph-client.admin.54398.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67667.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32037.log.gz -- replaced with /var/log/ceph/ceph-client.admin.52838.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69618.log 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49431.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64546.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48443.log: /var/log/ceph/ceph-client.admin.58060.log: /var/log/ceph/ceph-client.admin.50523.log: /var/log/ceph/ceph-client.admin.51694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48443.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58060.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51694.log.gz 0.0% 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50523.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56406.log.gz 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40499.log 2024-09-16T02:55:26.347 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69618.log.gz 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69722.log 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67277.log 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40499.log.gz 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52266.log 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69722.log.gz 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67277.log.gz 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54945.log 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52266.log.gz 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41909.log 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70294.log 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54945.log.gz 2024-09-16T02:55:26.348 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61216.log 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41909.log.gz 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46959.log 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70294.log.gz 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40811.log 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61216.log.gz 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53722.log 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46959.log.gz 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69071.log 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40811.log.gz 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53852.log 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53722.log.gz 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69071.log.gz 2024-09-16T02:55:26.349 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38990.log 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53852.log.gz 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67719.log 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70060.log 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38990.log.gz 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67433.log 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67719.log.gz 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69930.log 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70060.log.gz 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66315.log 0.0% 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67433.log.gz 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64989.log 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69930.log.gz 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54476.log 2024-09-16T02:55:26.350 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66315.log.gz 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51381.log 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64989.log.gz 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44121.log 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54476.log.gz 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40681.log 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51381.log.gz 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64077.log 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44121.log.gz 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51329.log 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40681.log.gz 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64077.log.gz 2024-09-16T02:55:26.351 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49899.log 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51329.log.gz 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68863.log 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48807.log 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49899.log.gz 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57337.log 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68863.log.gz 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32602.log 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48807.log.gz 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65015.log 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57337.log.gz 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45841.log 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32602.log.gz 2024-09-16T02:55:26.352 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61164.log 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65015.log.gz 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47011.log 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45841.log.gz 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61164.log.gz 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55468.log 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47011.log.gz 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63219.log 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39120.log 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55468.log.gz 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55049.log 2024-09-16T02:55:26.353 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63219.log.gz 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46283.log 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39120.log.gz 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66523.log 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55049.log.gz 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38262.log 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46283.log.gz 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31698.log 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66523.log.gz 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68109.log 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38262.log.gz 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35896.log 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31698.log.gz 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64833.log 2024-09-16T02:55:26.354 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68109.log.gz 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44642.log 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35896.log.gz 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33320.log 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64833.log.gz 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62465.log 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44642.log.gz 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55626.log 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33320.log.gz 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47271.log 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62465.log.gz 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37300.log 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55626.log.gz 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42924.log 2024-09-16T02:55:26.355 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47271.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34310.log 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37300.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44017.log 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42924.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53956.log 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34310.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44017.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47376.log 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53956.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66133.log 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33632.log 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47376.log.gz 2024-09-16T02:55:26.356 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66133.log.gz 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62985.log 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62777.log 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33632.log.gz 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51642.log 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62985.log.gz 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39953.log 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62777.log.gz 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48417.log 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51642.log.gz 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53514.log 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39953.log.gz 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48417.log.gz 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32042.log 2024-09-16T02:55:26.357 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68343.log 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53514.log.gz 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39615.log 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32042.log.gz 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39537.log 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68343.log.gz 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42716.log 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39615.log.gz 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36234.log 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39537.log.gz 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44879.log 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42716.log.gz 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48989.log 2024-09-16T02:55:26.358 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36234.log.gz 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55179.log 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44879.log.gz 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55808.log 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48989.log.gz 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37586.log 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55179.log.gz 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42118.log 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55808.log.gz 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38522.log 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37586.log.gz 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53124.log 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42118.log.gz 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38782.log 2024-09-16T02:55:26.359 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38522.log.gz 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49561.log 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53124.log.gz 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38782.log.gz 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34206.log 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60852.log 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49561.log.gz 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48833.log 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34206.log.gz 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.60852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33346.log 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60852.log.gz 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48833.log.gz 2024-09-16T02:55:26.360 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50731.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33346.log.gz 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26507.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55782.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50731.log.gz 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26507.log.gz 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32301.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36728.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55782.log.gz 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32301.log.gz 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47740.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32502.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36728.log.gz 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48469.log 2024-09-16T02:55:26.361 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47740.log.gz 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52656.log 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32502.log.gz 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38574.log 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48469.log.gz 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47922.log 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52656.log.gz 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38574.log.gz 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32406.log 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47922.log.gz 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56432.log 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45061.log 2024-09-16T02:55:26.362 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32406.log.gz 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52890.log 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56432.log.gz 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33998.log 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45061.log.gz 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33268.log 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52890.log.gz 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40655.log 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33998.log.gz 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66055.log 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33268.log.gz 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40655.log.gz 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61684.log 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47245.log 2024-09-16T02:55:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66055.log.gz 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39849.log 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61684.log.gz 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39328.log 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47245.log.gz 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62621.log 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39849.log.gz 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42950.log 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39328.log.gz 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37170.log 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62621.log.gz 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36260.log 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42950.log.gz 2024-09-16T02:55:26.364 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45737.log 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37170.log.gz 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55387.log 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36260.log.gz 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45737.log.gz 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46647.log 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48781.log 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55387.log.gz 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66575.log 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46647.log.gz 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67199.log 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48781.log.gz 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34362.log 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66575.log.gz 2024-09-16T02:55:26.365 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67199.log.gz 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42794.log 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34362.log.gz 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41831.log 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54372.log 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42794.log.gz 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61398.log 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41831.log.gz 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37820.log 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54372.log.gz 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61398.log.gz 2024-09-16T02:55:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46751.log 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37820.log.gz 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70112.log 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38964.log 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46751.log.gz 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31834.log 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70112.log.gz 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38964.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.38964.log.gz -- 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr: /var/log/ceph/ceph-client.admin.53332.log 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52864.log 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31834.log.gz 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63115.log 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53332.log.gz 2024-09-16T02:55:26.367 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52864.log.gz 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66939.log 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49301.log 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63115.log.gz 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70346.log 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66939.log.gz 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49301.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49301.log.gz 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54684.log 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68187.log 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70346.log.gz 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57882.log 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54684.log.gz 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63869.log 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68187.log.gz 2024-09-16T02:55:26.368 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54008.log 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57882.log.gz 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36286.log 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63869.log.gz 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54008.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.54294.log 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54008.log.gz 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50419.log 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36286.log.gz 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67589.log 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54294.log.gz 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50419.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.50419.log.gz --verbose 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr: -- /var/log/ceph/ceph-client.admin.46621.log 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32212.log 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67589.log.gz 2024-09-16T02:55:26.369 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68551.log 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46621.log.gz 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32212.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.32212.log.gz -- 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr: /var/log/ceph/ceph-client.admin.39563.log 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43289.log 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68551.log.gz 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39458.log 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39563.log.gz 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64364.log 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43289.log.gz 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56016.log 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39458.log.gz 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68941.log 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64364.log.gz 2024-09-16T02:55:26.370 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70554.log 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56016.log.gz 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61086.log 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68941.log.gz 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53436.log 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70554.log.gz 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38054.log 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61086.log.gz 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62231.log 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53436.log.gz 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51980.log 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38054.log.gz 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62231.log.gz 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68603.log 2024-09-16T02:55:26.371 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66757.log 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51980.log.gz 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68603.log.gz 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36962.log 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66757.log.gz 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40083.log 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gz 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49951.log 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36962.log.gz 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39875.log 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40083.log.gz 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33216.log 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49951.log.gz 2024-09-16T02:55:26.372 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62673.log 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39875.log.gz 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62803.log 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33216.log.gz 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66263.log 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62673.log.gz 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63843.log 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62803.log.gz 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43133.log 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66263.log.gz 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32238.log 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63843.log.gz 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62517.log 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43133.log.gz 2024-09-16T02:55:26.373 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56224.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32238.log.gz 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50029.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62517.log.gz 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68317.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56224.log.gz 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57729.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50029.log.gz 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64390.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68317.log.gz 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43367.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57729.log.gz 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52916.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64390.log.gz 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32501.log 2024-09-16T02:55:26.374 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43367.log.gz 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58246.log 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52916.log.gz 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41414.log 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32501.log.gz 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63739.log 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58246.log.gz 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69227.log 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41414.log.gz 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46933.log 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63739.log.gz 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41206.log 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69227.log.gz 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67459.log 2024-09-16T02:55:26.375 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46933.log.gz 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35454.log 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41206.log.gz 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65587.log 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67459.log.gz 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63687.log 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35454.log.gz 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65587.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.40785.log 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65587.log.gz 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55938.log 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63687.log.gz 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56822.log 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40785.log.gz 2024-09-16T02:55:26.376 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55938.log.gz 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52318.log 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38444.log 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56822.log.gz 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62100.log 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52318.log.gz 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50003.log 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38444.log.gz 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69696.log 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62100.log.gz 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35558.log 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50003.log.gz 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41883.log 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69696.log.gz 2024-09-16T02:55:26.377 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45763.log 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35558.log.gz 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67875.log 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41883.log.gz 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54658.log 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45763.log.gz 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35506.log/var/log/ceph/ceph-client.admin.67875.log: 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67875.log.gz 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55023.log 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54658.log.gz 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26569.log 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35506.log.gz 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63895.log 2024-09-16T02:55:26.378 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55023.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45009.log 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26569.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42690.log 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63895.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57623.log 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45009.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40265.log 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42690.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42196.log 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57623.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47610.log 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40265.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49067.log 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42196.log.gz 2024-09-16T02:55:26.379 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67563.log 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47610.log.gz 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49067.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.49067.log.gz /var/log/ceph/ceph-client.admin.40447.log 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68915.log 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67563.log.gz 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67381.log 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40447.log.gz 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69331.log 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68915.log.gz 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45867.log 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67381.log.gz 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43471.log 2024-09-16T02:55:26.380 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69331.log.gz 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58086.log 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45867.log.gz 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61034.log 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43471.log.gz 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65223.log 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58086.log.gz 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35870.log 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61034.log.gz 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51173.log 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65223.log.gz 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40603.log 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35870.log.gz 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35766.log 2024-09-16T02:55:26.381 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51173.log.gz 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66705.log 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55127.log 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35766.log.gz 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26256.log 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66705.log.gz 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66601.log 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55127.log.gz 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63583.log 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26256.log.gz 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57359.log 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66601.log.gz 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52500.log 2024-09-16T02:55:26.382 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63583.log.gz 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56640.log 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57359.log.gz 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34960.log 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52500.log.gz 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56640.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56640.log.gz 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35818.log 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34960.log.gz 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69357.log 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66185.log 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35818.log.gz 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69357.log.gzgzip 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43341.log 2024-09-16T02:55:26.383 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66185.log.gz 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64207.log 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32803.log 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43341.log.gz 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64207.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64207.log.gz 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46829.log 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69305.log 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32803.log.gz 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46829.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46829.log.gz 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62257.log 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39745.log 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69305.log.gz 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62257.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.62257.log.gz /var/log/ceph/ceph-client.admin.48625.log 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.384 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63505.log 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39745.log.gz 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49223.log 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48625.log.gz 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63505.log.gz 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49223.log.gz 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33034.log 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51225.log 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33034.log.gz 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44590.log 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49691.log 2024-09-16T02:55:26.385 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51225.log.gz 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44590.log.gz 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56952.log 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68811.log 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49691.log.gz 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56952.log.gz 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38184.log 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68993.log 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68811.log.gz 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38184.log.gz 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54606.log 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49665.log 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68993.log.gz 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49353.log 2024-09-16T02:55:26.386 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54606.log.gz 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49665.log.gz 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49353.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.49353.log.gz /var/log/ceph/ceph-client.admin.42872.log 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58112.log 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42872.log.gz 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51824.log 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49171.log 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58112.log.gz 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51824.log.gz 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52552.log 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35532.log 2024-09-16T02:55:26.387 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49171.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52552.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61944.log 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57734.log 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35532.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61944.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68837.log 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49015.log 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57734.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45451.log 0.0% 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68837.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42508.log 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49015.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45451.log.gz 2024-09-16T02:55:26.388 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48208.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48339.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42508.log.gz 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48208.log.gz 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61840.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50211.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48339.log.gz 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61840.log.gz 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36884.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53176.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50211.log.gz 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36884.log.gz 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32509.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61762.log 2024-09-16T02:55:26.389 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53176.log.gz 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65847.log 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32509.log.gz 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61762.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61762.log.gz -5 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36442.log 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65847.log.gz 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64181.log 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43054.log 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36442.log.gz 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64181.log.gz 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44200.log 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69670.log 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43054.log.gz 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44200.log.gz 2024-09-16T02:55:26.390 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39068.log 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62907.log 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69670.log.gz 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39068.log.gz 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45945.log 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62907.log.gz 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45945.log.gz 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37768.log 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48391.log 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37768.log.gz 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48391.log.gzgzip 2024-09-16T02:55:26.391 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57004.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62335.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57004.log.gz 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49379.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52604.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62335.log.gz 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49379.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.49379.log.gz /var/log/ceph/ceph-client.admin.68057.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33086.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52604.log.gz 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68057.log.gz 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54502.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47870.log 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33086.log.gz 2024-09-16T02:55:26.392 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53618.log 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54502.log.gz 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62829.log 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47870.log.gz 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53618.log.gz 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57930.log 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68239.log 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62829.log.gz 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64963.log 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57930.log.gz 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70606.log 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68239.log.gz 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51433.log 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64963.log.gz 2024-09-16T02:55:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70606.log.gz 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70398.log 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51277.log 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51433.log.gz 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62074.log 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70398.log.gz 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36598.log 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51277.log.gz 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53904.log 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62074.log.gz 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58320.log 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36598.log.gz 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34102.log 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53904.log.gz 2024-09-16T02:55:26.394 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37690.log 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58320.log.gz 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44460.log 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34102.log.gz 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48261.log 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37690.log.gz 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53540.log 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44460.log.gz 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48261.log.gzgzip 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50471.log 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47349.log 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53540.log.gz 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43159.log 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50471.log.gz 2024-09-16T02:55:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65119.log 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47349.log.gz 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54086.log 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43159.log.gz 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37274.log 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65119.log.gz 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53592.log 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54086.log.gz 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46595.log 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37274.log.gz 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37950.log 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53592.log.gz 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43419.log 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46595.log.gz 2024-09-16T02:55:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70164.log 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37950.log.gz 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36052.log 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43419.log.gz 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49847.log 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70164.log.gz 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39146.log 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36052.log.gz 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36572.log 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49847.log.gz 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34232.log 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39146.log.gz 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53384.log 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36572.log.gz 2024-09-16T02:55:26.397 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39771.log 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34232.log.gz 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53384.log.gz 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38080.log 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39771.log.gz 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46049.log 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47115.log 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38080.log.gz 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34648.log 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46049.log.gz 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64676.log 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47115.log.gz 2024-09-16T02:55:26.398 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37144.log 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34648.log.gz 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64676.log.gz 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42612.log 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37144.log.gz 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64703.log 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63713.log 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42612.log.gz 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44746.log 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64703.log.gz 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63713.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.63713.log.gz 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr: /var/log/ceph/ceph-client.admin.56614.log 2024-09-16T02:55:26.399 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49093.log 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44746.log.gz 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66471.log 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56614.log.gz 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68291.log 0.0% 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49093.log.gz 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47402.log 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66471.log.gz 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55205.log 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68291.log.gz 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66367.log 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47402.log.gz 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45815.log 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55205.log.gz 2024-09-16T02:55:26.400 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40161.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66367.log.gz 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70372.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45815.log.gz 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34856.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40161.log.gz 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68005.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70372.log.gz 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32395.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34856.log.gz 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34726.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68005.log.gz 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57134.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32395.log.gz 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50757.log 2024-09-16T02:55:26.401 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34726.log.gz 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64051.log 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57134.log.gz 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68525.log 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50757.log.gz 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35402.log 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64051.log.gz 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68525.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.68525.log.gz 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45087.log 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69514.log 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35402.log.gz 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52006.log 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45087.log.gz 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57579.log 2024-09-16T02:55:26.402 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69514.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53072.log 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52006.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33790.log 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57579.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39641.log 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53072.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70580.log 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33790.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37560.log 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39641.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55704.log 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70580.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36078.log 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37560.log.gz 2024-09-16T02:55:26.403 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65145.log 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55704.log.gz 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36208.log 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36078.log.gz 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68577.log 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65145.log.gz 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36390.log 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36208.log.gz 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44538.log 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68577.log.gz 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54190.log 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36390.log.gz 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61814.log 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44538.log.gz 2024-09-16T02:55:26.404 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43028.log 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54190.log.gz 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32930.log 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61814.log.gz 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47480.log 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43028.log.gz 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32930.log.gz 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55964.log 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47480.log.gz 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54242.log 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38418.log 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55964.log.gz 2024-09-16T02:55:26.405 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54242.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.38652.log 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54242.log.gz 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38418.log.gz 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61112.log 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61606.log 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38652.log.gz 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67511.log 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61112.log.gz 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39979.log 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61606.log.gz 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67511.log.gz 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50497.log 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39979.log.gz 2024-09-16T02:55:26.406 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43627.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43913.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50497.log.gz 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44827.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43627.log.gz 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43913.log.gz 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69800.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44827.log.gz 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32299.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33868.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69800.log.gz 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32038.log 2024-09-16T02:55:26.407 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32299.log.gz 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63817.log 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33868.log.gz 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64259.log 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32038.log.gz 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63947.log 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63817.log.gz 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45347.log 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64259.log.gz 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64103.log 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63947.log.gz 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37976.log 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45347.log.gz 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2024-09-16T02:55:26.408 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64103.log.gz 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38756.log 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37976.log.gz 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42170.log 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31796.log 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38756.log.gz 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47974.log 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42170.log.gz 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66627.log 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31796.log.gz 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.47974.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63349.log 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47974.log.gz 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60982.log 2024-09-16T02:55:26.409 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66627.log.gz 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51850.log 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63349.log.gz 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.60982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64859.log 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60982.log.gz 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55652.log 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51850.log.gz 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69461.log 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64859.log.gz 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54320.log 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55652.log.gz 2024-09-16T02:55:26.410 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53046.log 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69461.log.gz 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41180.log 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54320.log.gz 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55548.log 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53046.log.gz 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37872.log 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41180.log.gz 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37404.log 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55548.log.gz 2024-09-16T02:55:26.413 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70320.log 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37872.log.gz 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62439.log 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37404.log.gz 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62595.log 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70320.log.gz 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56458.log 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62439.log.gz 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61424.log 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62595.log.gz 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67797.log 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56458.log.gz 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33764.log 2024-09-16T02:55:26.414 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61424.log.gz 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53488.log 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67797.log.gz 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68369.log 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33764.log.gz 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65353.log 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53488.log.gz 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32510.log 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68369.log.gz 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47818.log 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65353.log.gz 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34908.log 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32510.log.gz 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51668.log 2024-09-16T02:55:26.415 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47818.log.gz 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.34908.log: -- /var/log/ceph/ceph-client.admin.34622.log 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34908.log.gz 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56120.log 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51668.log.gz 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63011.log 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34622.log.gz 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41440.log 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56120.log.gz 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69904.log 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63011.log.gz 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45789.log 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41440.log.gz 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42664.log 2024-09-16T02:55:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69904.log.gz 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65379.log 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45789.log.gz 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52448.log 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42664.log.gz 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32030.log 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65379.log.gz 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37846.log 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52448.log.gz 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32030.log.gz 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34414.log 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37846.log.gz 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57535.log 2024-09-16T02:55:26.417 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34518.log 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34414.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54710.log 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57535.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33842.log 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34518.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63661.log 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54710.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50185.log 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33842.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35792.log 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63661.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35324.log 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50185.log.gz 2024-09-16T02:55:26.418 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62647.log 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35792.log.gz 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50133.log 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35324.log.gz 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51720.log 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62647.log.gz 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52786.log 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50133.log.gz 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61710.log 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51720.log.gz 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38496.log 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52786.log.gz 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69175.log 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61710.log.gz 2024-09-16T02:55:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49041.log 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38496.log.gz 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55574.log 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69175.log.gz 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47323.log 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49041.log.gz 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35116.log 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55574.log.gz 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44853.log 0.0% 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47323.log.gz 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56354.log 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35116.log.gz 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42300.log 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44853.log.gz 2024-09-16T02:55:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58008.log 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56354.log.gz 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68213.log 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42300.log.gz 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50965.log 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58008.log.gz 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43965.log 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68213.log.gz 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31744.log 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50965.log.gz 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40998.log 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43965.log.gz 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31744.log.gz 2024-09-16T02:55:26.421 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70086.log 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40998.log.gz 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34284.log 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40109.log.gz 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42326.log 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70086.log.gz 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69279.log 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34284.log.gz 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46179.log 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42326.log.gz 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31673.log 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69279.log.gz 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67537.log 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46179.log.gz 2024-09-16T02:55:26.422 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45685.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31673.log.gz 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32706.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67537.log.gz 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54346.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45685.log.gz 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58242.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32706.log.gz 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41362.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54346.log.gz 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70502.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58242.log.gz 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67485.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41362.log.gz 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50887.log 2024-09-16T02:55:26.423 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70502.log.gz 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63037.log 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67485.log.gz 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43575.log 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50887.log.gz 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40577.log 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63037.log.gz 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52396.log 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43575.log.gz 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32904.log 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40577.log.gz 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56926.log 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52396.log.gz 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44801.log 2024-09-16T02:55:26.424 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32904.log.gz 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63999.log 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56926.log.gz 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44801.log.gz 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47037.log 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56666.log 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63999.log.gz 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50809.log 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47037.log.gz 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63791.log 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56666.log.gz 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31833.log 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50809.log.gz 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55075.log 0.0% 2024-09-16T02:55:26.425 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63791.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64416.log 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31833.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46699.log 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55075.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35350.log 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64416.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66159.log 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46699.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34050.log 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35350.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36936.log 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66159.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34050.log.gz 2024-09-16T02:55:26.426 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36936.log.gz 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55309.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35688.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53462.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55309.log.gz 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40395.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35688.log.gz 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51616.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53462.log.gz 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46023.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40395.log.gz 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61736.log 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51616.log.gz 2024-09-16T02:55:26.427 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69253.log 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46023.log.gz 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65275.log 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61736.log.gz 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62933.log 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69253.log.gz 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36494.log 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65275.log.gz 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67979.log 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62933.log.gz 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36156.log 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36494.log.gz 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50159.log 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67979.log.gz 2024-09-16T02:55:26.428 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36156.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.41779.log 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36156.log.gz 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50159.log.gz 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64937.log 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43185.log 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44694.log 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64937.log.gz 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61502.log 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43185.log.gz 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44069.log 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44694.log.gz 2024-09-16T02:55:26.429 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55257.log 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61502.log.gz 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.44069.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63401.log 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44069.log.gz 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57425.log 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55257.log.gz 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32956.log 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63401.log.gz 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35948.log 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57425.log.gz 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61632.log 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32956.log.gz 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67329.log 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35948.log.gz 2024-09-16T02:55:26.430 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61632.log.gz 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35662.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67329.log.gz 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34700.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48078.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35662.log.gz 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47558.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34700.log.gz 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65171.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48078.log.gz 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64468.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47558.log.gz 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48599.log 2024-09-16T02:55:26.431 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65171.log.gz 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67017.log 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64468.log.gz 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64885.log 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48599.log.gz 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56068.log 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67017.log.gz 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42092.log 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64885.log.gz 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47141.log 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56068.log.gz 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45893.log 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42092.log.gz 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70268.log 2024-09-16T02:55:26.432 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47141.log.gz 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43939.log 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45893.log.gz 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35844.log 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70268.log.gz 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54138.log 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43939.log.gz 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54893.log 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35844.log.gz 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69826.log 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54138.log.gz 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54893.log.gz 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67927.log 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42404.log 2024-09-16T02:55:26.433 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69826.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41988.log 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67927.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65561.log 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42404.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37092.log 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41988.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46127.log 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65561.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67693.log 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37092.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55231.log 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46127.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38132.log 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67693.log.gz 2024-09-16T02:55:26.434 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69149.log 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55231.log.gz 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68499.log 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38132.log.gz 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66341.log 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69149.log.gz 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68655.log 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68499.log.gz 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33372.log 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66341.log.gz 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57730.log 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68655.log.gz 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57670.log 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33372.log.gz 2024-09-16T02:55:26.435 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43601.log 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57730.log.gz 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46153.log 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57670.log.gz 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50107.log 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43601.log.gz 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46153.log.gz 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38860.log 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50107.log.gz 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.39016.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43211.log 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54450.log 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38860.log.gz 2024-09-16T02:55:26.436 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54164.log 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43211.log.gz 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:gzip/var/log/ceph/ceph-client.admin.54450.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67901.log 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54450.log.gz 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44512.log 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54164.log.gz 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51407.log 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67901.log.gz 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44512.log.gz 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42820.log 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51407.log.gz 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37898.log 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2024-09-16T02:55:26.437 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50081.log 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42820.log.gz 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50055.log 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37898.log.gz 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66107.log 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50081.log.gz 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36624.log 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50055.log.gz 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51251.log 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66107.log.gz 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54867.log 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36624.log.gz 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56536.log 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51251.log.gz 2024-09-16T02:55:26.438 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54867.log.gz 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47454.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56536.log.gz 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48651.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38366.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47454.log.gz 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48651.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48651.log.gz 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66835.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38366.log.gz 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53280.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62881.log 2024-09-16T02:55:26.439 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66835.log.gz 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36780.log 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53280.log.gz 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62881.log: /var/log/ceph/ceph-client.admin.36780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42352.log 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62881.log.gz 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36780.log.gz 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32264.log 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34804.log 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42352.log.gz 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33658.log 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32264.log.gz 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33138.log 2024-09-16T02:55:26.440 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34804.log: /var/log/ceph/ceph-client.admin.33658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34804.log.gz 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33658.log.gz 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42560.log 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61138.log 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33138.log.gz 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61970.log 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42560.log.gz 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49977.log 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61138.log.gz 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61970.log.gz 2024-09-16T02:55:26.441 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57315.log 2024-09-16T02:55:26.548 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52110.log 2024-09-16T02:55:26.548 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49977.log.gz 2024-09-16T02:55:26.548 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55494.log 2024-09-16T02:55:26.548 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57315.log.gz 2024-09-16T02:55:26.548 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38548.log 2024-09-16T02:55:26.548 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52110.log.gz 2024-09-16T02:55:26.548 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34076.log 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55494.log: /var/log/ceph/ceph-client.admin.38548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63479.log 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55494.log.gz 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38548.log.gz 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34076.log.gz 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35376.log 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48755.log 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63479.log: /var/log/ceph/ceph-client.admin.35376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38314.log 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35376.log.gz 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63479.log.gz 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48755.log.gz 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44095.log 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57731.log 2024-09-16T02:55:26.549 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38314.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64285.log 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44095.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40239.log 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57731.log: /var/log/ceph/ceph-client.admin.64285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57731.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64285.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40239.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52084.log 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52370.log 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36520.log 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52084.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52370.log.gz 2024-09-16T02:55:26.550 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66809.log 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70633.log 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36520.log: /var/log/ceph/ceph-client.admin.66809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36520.log.gz 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36702.log 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66809.log.gz 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70633.log.gz 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36806.log 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41154.log 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36702.log.gz 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66029.log 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36806.log.gz 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56718.log 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41154.log: /var/log/ceph/ceph-client.admin.66029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65899.log 0.0% 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41154.log.gz 2024-09-16T02:55:26.551 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66029.log.gz 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56718.log.gz 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39485.log 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42976.log 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65899.log.gz 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44226.log 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39485.log.gz 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35298.log 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42976.log: /var/log/ceph/ceph-client.admin.44226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34388.log 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42976.log.gz 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44226.log.gz 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35298.log.gz 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44616.log 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55834.log 2024-09-16T02:55:26.552 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34388.log: /var/log/ceph/ceph-client.admin.44616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32708.log 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34388.log.gz 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44616.log.gz 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55834.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.55834.log.gz 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32852.log 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64755.log 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32708.log: /var/log/ceph/ceph-client.admin.32852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37742.log 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32708.log.gz 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32852.log.gz 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64755.log.gz 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37612.log 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44356.log 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37742.log.gz/var/log/ceph/ceph-client.admin.37612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62022.log 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.553 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37612.log.gz 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44356.log.gz 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46803.log 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36650.log 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62022.log.gz 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45321.log 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46803.log.gz 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43106.log 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36650.log: /var/log/ceph/ceph-client.admin.45321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63271.log 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36650.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45321.log.gz 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43106.log.gz 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-09-16T02:55:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42456.log 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63271.log.gz 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38210.log 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43393.log 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38210.log: /var/log/ceph/ceph-client.admin.42456.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54112.log 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38210.log.gz 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42456.log.gz 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43393.log.gz 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61918.log 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34336.log 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54112.log.gz 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29988.log 2024-09-16T02:55:26.555 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61918.log.gz 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34336.log.gz 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44931.log 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63063.log 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.29988.log: /var/log/ceph/ceph-client.admin.44931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53774.log 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29988.log.gz 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44931.log.gz 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63063.log.gz 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55416.log 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56484.log 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53774.log.gz 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50601.log 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55416.log.gz 2024-09-16T02:55:26.556 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47506.log 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56484.log: /var/log/ceph/ceph-client.admin.50601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56484.log.gz 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56588.log 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50601.log.gz 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47506.log.gz 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42742.log 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56588.log.gz 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62569.log 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34154.log 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42742.log: /var/log/ceph/ceph-client.admin.62569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45139.log 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42742.log.gz 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62569.log.gz 2024-09-16T02:55:26.557 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34154.log.gz 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62543.log 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45139.log.gz 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36364.log 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62543.log.gz 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50835.log 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65327.log 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36364.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50835.log: /var/log/ceph/ceph-client.admin.37352.log 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36364.log.gz 0.0% 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50835.log.gz 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65327.log.gz 2024-09-16T02:55:26.558 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64129.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68421.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37352.log.gz 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43705.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64129.log.gz 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63557.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68421.log: /var/log/ceph/ceph-client.admin.43705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68733.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43705.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68421.log.gz 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63557.log.gz 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33502.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70138.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68733.log: /var/log/ceph/ceph-client.admin.33502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66289.log 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68733.log.gz 2024-09-16T02:55:26.559 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33502.log.gz 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70138.log.gz 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54971.log 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46101.log 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66289.log: /var/log/ceph/ceph-client.admin.54971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39927.log 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54971.log.gz 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66289.log.gz 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46101.log.gz 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69201.log 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54788.log 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39927.log.gz 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48130.log 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69201.log.gz 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34544.log 2024-09-16T02:55:26.560 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54788.log: /var/log/ceph/ceph-client.admin.48130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52760.log 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54788.log.gz 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48130.log.gz 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34544.log.gz 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54814.log 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43237.log 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52760.log: /var/log/ceph/ceph-client.admin.54814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48495.log 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52760.log.gz 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54814.log.gz 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43237.log.gz 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57271.log 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44382.log 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48495.log: /var/log/ceph/ceph-client.admin.57271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48885.log 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48495.log.gz 2024-09-16T02:55:26.561 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57271.log.gz 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44382.log.gz 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61346.log 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41805.log 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48885.log.gz 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37222.log 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61346.log.gz 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64338.log 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41805.log: /var/log/ceph/ceph-client.admin.37222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41805.log.gz 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37222.log.gz 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47688.log 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35246.log 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64338.log.gz 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34466.log 2024-09-16T02:55:26.562 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47688.log.gz 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56302.log 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38886.log 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34466.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35246.log.gz 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56302.log.gz 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49483.log 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46231.log 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38886.log.gz 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44174.log 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49483.log.gz 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50367.log 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46231.log: /var/log/ceph/ceph-client.admin.44174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61294.log 2024-09-16T02:55:26.563 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46231.log.gz 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44174.log.gz 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50367.log.gz 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31828.log 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69852.log 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61294.log.gz 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61372.log 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31828.log.gz 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69852.log: /var/log/ceph/ceph-client.admin.61372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58141.log 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69852.log.gz 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61372.log.gz 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51928.log 2024-09-16T02:55:26.564 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45711.log 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58141.log.gz 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50445.log 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51928.log: /var/log/ceph/ceph-client.admin.45711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51928.log.gz 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45711.log.gz 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46205.log 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68083.log 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50445.log.gz 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46205.log.gz 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65041.log 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55990.log 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68083.log.gz 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36182.log 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65041.log.gz 2024-09-16T02:55:26.565 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55990.log.gz 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66003.log 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36182.log.gz 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65301.log 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67615.log 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66003.log: /var/log/ceph/ceph-client.admin.65301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66003.log.gz 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65301.log.gz 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52422.log 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63089.log 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67615.log.gz 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52422.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39511.log 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63089.log.gz 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50783.log 2024-09-16T02:55:26.566 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33476.log 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39511.log.gz 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50783.log.gz 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42378.log 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35610.log 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33476.log.gz 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42378.log.gz 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65457.log 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26531.log 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35610.log.gz 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65457.log.gz 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35922.log 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32628.log 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26531.log: /var/log/ceph/ceph-client.admin.35922.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26531.log.gz 2024-09-16T02:55:26.567 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35922.log.gz 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50393.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40629.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32628.log.gz 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50393.log.gz 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47662.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40629.log.gz 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49873.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47662.log: /var/log/ceph/ceph.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47662.log.gz 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65977.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49873.log.gz 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57209.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65977.log.gz 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33450.log 2024-09-16T02:55:26.568 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57209.log: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.57209.log.gz 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66393.log 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33450.log.gz 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47792.log 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66393.log.gz 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53982.log 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47792.log.gz 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51121.log 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53982.log.gz 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62204.log 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51121.log.gz 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63973.log 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62204.log.gz 2024-09-16T02:55:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62152.log 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63973.log.gz 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr: 96.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32982.log 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62152.log.gz 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70008.log 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44564.log 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32982.log: /var/log/ceph/ceph-client.admin.70008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32982.log.gz 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70008.log.gz 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51772.log 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49249.log 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44564.log: /var/log/ceph/ceph-client.admin.51772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44564.log.gz 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51772.log.gz 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46413.log 2024-09-16T02:55:26.570 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67043.log 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49249.log: /var/log/ceph/ceph-client.admin.46413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49249.log.gz 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46413.log.gz 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51746.log 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67043.log: /var/log/ceph/ceph-client.admin.38288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67043.log.gz 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41726.log 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57447.log 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51746.log: /var/log/ceph/ceph-client.admin.41726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51746.log.gz 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41726.log.gz 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52630.log 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47584.log 2024-09-16T02:55:26.571 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57447.log: /var/log/ceph/ceph-client.admin.52630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57447.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52630.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45919.log 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47584.log: /var/log/ceph/ceph-client.admin.39406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47584.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69956.log 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65925.log 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45919.log: /var/log/ceph/ceph-client.admin.69956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45919.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69956.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38236.log 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33008.log 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65925.log: /var/log/ceph/ceph-client.admin.38236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65925.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38236.log.gz 2024-09-16T02:55:26.572 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63245.log 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57293.log 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33008.log: /var/log/ceph/ceph-client.admin.63245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33008.log.gz 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63245.log.gz 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41518.log 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63167.log 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57293.log: /var/log/ceph/ceph-client.admin.41518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57293.log.gz 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41518.log.gz 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49119.log 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53644.log 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63167.log: /var/log/ceph/ceph-client.admin.49119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63167.log.gz 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49119.log.gz 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54424.log 2024-09-16T02:55:26.573 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33894.log 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53644.log: /var/log/ceph/ceph-client.admin.54424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53644.log.gz 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54424.log.gz 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52240.log 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49197.log 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33894.log: /var/log/ceph/ceph-client.admin.52240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33894.log.gz 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52240.log.gz 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48104.log 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37040.log 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49197.log.gz 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48104.log.gz 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57956.log 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37040.log.gz 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35038.log 2024-09-16T02:55:26.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36026.log 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57956.log.gz 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35038.log.gz 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36416.log 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36026.log.gz 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66887.log 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48365.log 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36416.log.gz 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66887.log.gz 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37456.log 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66783.log 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48365.log.gz 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37456.log.gz 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62725.log 2024-09-16T02:55:26.575 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66783.log.gz 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32856.log 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48026.log 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62725.log: /var/log/ceph/ceph-client.admin.32856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62725.log.gz 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32856.log.gz 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69097.log 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44905.log 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48026.log.gz 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69097.log.gz 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51355.log 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44905.log.gz 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51147.log 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60878.log 2024-09-16T02:55:26.576 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51355.log: /var/log/ceph/ceph-client.admin.51147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51355.log.gz 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51147.log.gz 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66445.log 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.60878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60878.log.gz 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66445.log.gz 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61008.log 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64520.log 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53930.log 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61008.log.gz 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64520.log.gz 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62048.log 2024-09-16T02:55:26.577 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53930.log.gz 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38470.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57557.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62048.log.gz 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38470.log.gz 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53228.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57557.log.gz 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69566.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54632.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53228.log.gz 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69566.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63323.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69566.log.gz 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54632.log.gz 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45607.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33112.log 2024-09-16T02:55:26.578 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63323.log.gz 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45607.log.gz 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53696.log 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33112.log.gz 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37924.log 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66965.log 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53696.log: /var/log/ceph/ceph-client.admin.37924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53696.log.gz 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37924.log.gz 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64624.log 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57878.log 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66965.log: /var/log/ceph/ceph-client.admin.64624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66965.log.gz 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64624.log.gz 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40889.log 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38392.log 2024-09-16T02:55:26.579 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57878.log.gz 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55600.log 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38392.log.gz 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46673.log 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41752.log 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55600.log.gz 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46673.log.gz 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68707.log 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65743.log 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41752.log.gz 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68707.log.gz 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64781.log 2024-09-16T02:55:26.580 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55283.log 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65743.log.gz 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64781.log.gz 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56874.log 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55283.log.gz 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39797.log 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38938.log 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56874.log: /var/log/ceph/ceph-client.admin.39797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56874.log.gz 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39797.log.gz 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61320.log 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69748.log 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38938.log.gz 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61320.log.gz 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63635.log 2024-09-16T02:55:26.581 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69748.log.gz 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57982.log 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56770.log 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63635.log: /var/log/ceph/ceph-client.admin.57982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63635.log.gz 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57982.log.gz 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36000.log 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53670.log 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56770.log.gz 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36000.log.gz 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50627.log 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53670.log.gz 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33294.log 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58142.log 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50627.log.gz 2024-09-16T02:55:26.582 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33294.log.gz 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62855.log 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58142.log.gz 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56744.log 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32398.log 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62855.log.gz 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56744.log.gz 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63141.log 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32398.log.gz 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51537.log 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63765.log 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63141.log.gz 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51537.log.gz 2024-09-16T02:55:26.583 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53098.log 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63765.log.gz 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32707.log 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49795.log 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53098.log.gz 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32707.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45997.log 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49795.log.gz 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68031.log 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49743.log 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45997.log.gz 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68031.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65509.log 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68031.log.gz 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49743.log.gz 2024-09-16T02:55:26.584 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64442.log 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35428.log 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65509.log: /var/log/ceph/ceph-client.admin.64442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65509.log.gz 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64442.log.gz 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65639.log 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65665.log 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35428.log: /var/log/ceph/ceph-client.admin.65639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35428.log.gz 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65639.log.gz 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53150.log 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67953.log 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65665.log: /var/log/ceph/ceph-client.admin.53150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65665.log.gz 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53150.log.gz 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2024-09-16T02:55:26.585 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55101.log 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67953.log.gz 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38808.log 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55101.log.gz 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45399.log 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52682.log 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38808.log.gz 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45399.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49327.log 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52682.log.gz 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49587.log 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65691.log 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49327.log.gz 2024-09-16T02:55:26.586 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49587.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70476.log 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65691.log.gz 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33816.log 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70476.log.gz 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37794.log 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33816.log.gz 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35740.log 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37794.log.gz 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48703.log 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35740.log.gz 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69982.log 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48703.log.gz 2024-09-16T02:55:26.587 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42040.log 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69982.log.gz 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50237.log 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42040.log.gz 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32186.log 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50237.log.gz 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61060.log 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32186.log.gz 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34778.log 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61060.log.gz 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56172.log 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34778.log.gz 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56380.log 2024-09-16T02:55:26.588 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56172.log.gz 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56146.log 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56380.log.gz 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51017.log 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56146.log.gz 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47219.log 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51017.log.gz 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47219.log.gz 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68681.log 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68681.log.gz 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45295.log 2024-09-16T02:55:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35636.log 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45295.log.gz 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61450.log 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35636.log.gz 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62361.log 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61450.log.gz 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35090.log 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62361.log.gz 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57830.log 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35090.log.gz 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63193.log 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57830.log.gz 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33528.log 2024-09-16T02:55:26.590 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63193.log.gz 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33528.log.gz 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52578.log 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65821.log 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52578.log.gz 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65821.log.gz 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44983.log 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35194.log 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44983.log.gz 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50575.log 2024-09-16T02:55:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35194.log.gz 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44043.log 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50575.log.gz 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58346.log 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44043.log.gz 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41857.log 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58346.log.gz 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41492.log 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41857.log.gz 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35714.log 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41492.log.gz 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64807.log 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35714.log.gz 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57491.log 2024-09-16T02:55:26.592 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64807.log.gz 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57491.log.gz 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45971.log 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57056.log 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45971.log.gz 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54528.log 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57056.log.gz 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46075.log 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54528.log.gz 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52214.log 2024-09-16T02:55:26.593 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46075.log.gz 2024-09-16T02:55:26.624 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33606.log 2024-09-16T02:55:26.624 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52214.log.gz 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32699.log 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33606.log.gz 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65483.log 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32699.log.gz 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62126.log 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65483.log.gz 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45555.log 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62126.log.gz 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54554.log 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45555.log.gz 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53878.log 2024-09-16T02:55:26.625 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54554.log.gz 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68629.log 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53878.log.gz 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45269.log 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68629.log.gz 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33684.log 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45269.log.gz 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63609.log 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33684.log.gz 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52526.log 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63609.log.gz 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46439.log 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52526.log.gz 2024-09-16T02:55:26.626 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67771.log 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46439.log.gz 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60904.log 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67771.log.gz 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34986.log 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.60904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60904.log.gz 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55860.log 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34986.log.gz 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70450.log 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55860.log.gz 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50289.log 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70450.log.gz 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33736.log 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50289.log.gz 2024-09-16T02:55:26.627 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39719.log 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33736.log.gz 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67407.log 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39719.log.gz 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34596.log 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67407.log.gz 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70689.log 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34596.log.gz 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45529.log 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70689.log.gz 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46361.log 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45529.log.gz 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34882.log 2024-09-16T02:55:26.628 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46361.log.gz 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39042.log 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34882.log.gz 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69435.log 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39042.log.gz 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42066.log 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69435.log.gz 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54060.log 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42066.log.gz 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37014.log 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54060.log.gz 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31648.log 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37014.log.gz 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57082.log 2024-09-16T02:55:26.629 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31648.log.gz 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46465.log 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57082.log.gz 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55335.log 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46465.log.gz 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67823.log 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55335.log.gz 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69487.log 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67823.log.gz 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54840.log 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69487.log.gz 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53202.log 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54840.log.gz 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33424.log 2024-09-16T02:55:26.630 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53202.log.gz 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62699.log 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33424.log.gz 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54216.log 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62699.log.gz 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50549.log 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54216.log.gz 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43080.log 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50549.log.gz 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40421.log 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43080.log.gz 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47844.log 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40421.log.gz 2024-09-16T02:55:26.631 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39693.log 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47844.log.gz 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70242.log 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39693.log.gz 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35064.log 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70242.log.gz 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36130.log 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35064.log.gz 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40317.log 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36130.log.gz 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33164.log 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40317.log.gz 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33164.log.gz 2024-09-16T02:55:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49535.log 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49457.log 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49535.log.gz 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69045.log 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49457.log.gz 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55522.log 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69045.log.gz 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49145.log 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55522.log.gz 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33554.log 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49145.log.gz 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62413.log 2024-09-16T02:55:26.633 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33554.log.gz 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44957.log 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62413.log.gz 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53410.log 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44957.log.gz 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46569.log 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53410.log.gz 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56510.log 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46569.log.gz 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47948.log 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56510.log.gz 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51511.log 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47948.log.gz 2024-09-16T02:55:26.634 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66991.log 2024-09-16T02:55:26.635 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51511.log.gz 2024-09-16T02:55:26.635 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T02:55:26.635 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66991.log.gz 2024-09-16T02:55:26.635 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33972.log 2024-09-16T02:55:26.642 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46257.log 2024-09-16T02:55:26.642 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33972.log.gz 2024-09-16T02:55:26.642 INFO:teuthology.orchestra.run.smithi137.stderr: 95.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T02:55:26.642 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47766.log 2024-09-16T02:55:26.642 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46257.log.gz 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47297.log 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47766.log.gz 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34258.log 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47297.log.gz 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57249.log 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34258.log.gz 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33398.log 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57249.log.gz 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63921.log 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33398.log.gz 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57601.log 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63921.log.gz 2024-09-16T02:55:26.643 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36754.log 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57601.log.gz 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70190.log 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36754.log.gz 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50861.log 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70190.log.gz 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34570.log 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50861.log.gz 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34570.log.gz 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45659.log 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56562.log 2024-09-16T02:55:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45659.log.gz 2024-09-16T02:55:26.645 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65769.log 2024-09-16T02:55:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56562.log.gz 2024-09-16T02:55:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48235.log 2024-09-16T02:55:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65769.log.gz 2024-09-16T02:55:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51954.log 2024-09-16T02:55:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48235.log.gz 2024-09-16T02:55:26.647 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58190.log 2024-09-16T02:55:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51954.log.gz 2024-09-16T02:55:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35012.log 2024-09-16T02:55:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58190.log.gz 2024-09-16T02:55:26.648 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52032.log 2024-09-16T02:55:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35012.log.gz 2024-09-16T02:55:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64572.log 2024-09-16T02:55:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52032.log.gz 2024-09-16T02:55:26.649 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65249.log 2024-09-16T02:55:26.650 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64572.log.gz 2024-09-16T02:55:26.650 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58034.log 2024-09-16T02:55:26.651 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65249.log.gz 2024-09-16T02:55:26.651 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34830.log 2024-09-16T02:55:26.652 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58034.log.gz 2024-09-16T02:55:26.652 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54997.log 2024-09-16T02:55:26.653 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34830.log.gz 2024-09-16T02:55:26.653 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70216.log 2024-09-16T02:55:26.654 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54997.log.gz 2024-09-16T02:55:26.654 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62751.log 2024-09-16T02:55:26.655 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70216.log.gz 2024-09-16T02:55:26.655 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54762.log 2024-09-16T02:55:26.656 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62751.log.gz 2024-09-16T02:55:26.656 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53566.log 2024-09-16T02:55:26.657 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54762.log.gz 2024-09-16T02:55:26.657 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43653.log 2024-09-16T02:55:26.658 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53566.log.gz 2024-09-16T02:55:26.658 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48573.log 2024-09-16T02:55:26.659 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43653.log.gz 2024-09-16T02:55:26.659 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40707.log 2024-09-16T02:55:26.660 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48573.log.gz 2024-09-16T02:55:26.660 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51590.log 2024-09-16T02:55:26.661 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40707.log.gz 2024-09-16T02:55:26.661 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62387.log 2024-09-16T02:55:26.662 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51590.log.gz 2024-09-16T02:55:26.662 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67121.log 2024-09-16T02:55:26.663 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62387.log.gz 2024-09-16T02:55:26.663 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53254.log 2024-09-16T02:55:26.664 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67121.log.gz 2024-09-16T02:55:26.664 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57826.log 2024-09-16T02:55:26.665 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53254.log.gz 2024-09-16T02:55:26.665 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43445.log 2024-09-16T02:55:26.666 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57826.log.gz 2024-09-16T02:55:26.666 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39224.log 2024-09-16T02:55:26.667 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43445.log.gz 2024-09-16T02:55:26.667 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48156.log 2024-09-16T02:55:26.668 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39224.log.gz 2024-09-16T02:55:26.668 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43835.log 2024-09-16T02:55:26.669 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48156.log.gz 2024-09-16T02:55:26.669 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52292.log 2024-09-16T02:55:26.670 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43835.log.gz 2024-09-16T02:55:26.670 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46335.log 2024-09-16T02:55:26.671 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52292.log.gz 2024-09-16T02:55:26.671 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38002.log 2024-09-16T02:55:26.672 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46335.log.gz 2024-09-16T02:55:26.672 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48729.log 2024-09-16T02:55:26.673 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38002.log.gz 2024-09-16T02:55:26.673 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48052.log 2024-09-16T02:55:26.674 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48729.log.gz 2024-09-16T02:55:26.674 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69774.log 2024-09-16T02:55:26.675 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48052.log.gz 2024-09-16T02:55:26.675 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42846.log 2024-09-16T02:55:26.676 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69774.log.gz 2024-09-16T02:55:26.676 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67225.log 2024-09-16T02:55:26.677 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42846.log.gz 2024-09-16T02:55:26.677 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47714.log 2024-09-16T02:55:26.678 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67225.log.gz 2024-09-16T02:55:26.679 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35142.log 2024-09-16T02:55:26.679 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47714.log.gz 2024-09-16T02:55:26.680 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66497.log 2024-09-16T02:55:26.680 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35142.log.gz 2024-09-16T02:55:26.681 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54580.log 2024-09-16T02:55:26.681 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66497.log.gz 2024-09-16T02:55:26.682 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63427.log 2024-09-16T02:55:26.683 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54580.log.gz 2024-09-16T02:55:26.683 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64911.log 2024-09-16T02:55:26.683 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63427.log.gz 2024-09-16T02:55:26.684 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57381.log 2024-09-16T02:55:26.684 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64911.log.gz 2024-09-16T02:55:26.685 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51199.log 2024-09-16T02:55:26.685 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57381.log.gz 2024-09-16T02:55:26.686 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39667.log 2024-09-16T02:55:26.686 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51199.log.gz 2024-09-16T02:55:26.687 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49769.log 2024-09-16T02:55:26.687 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39667.log.gz 2024-09-16T02:55:26.688 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60956.log 2024-09-16T02:55:26.688 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49769.log.gz 2024-09-16T02:55:26.689 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62309.log 2024-09-16T02:55:26.689 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.60956.log: 90.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 0.0% 2024-09-16T02:55:26.690 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60956.log.gz 2024-09-16T02:55:26.690 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67303.log 2024-09-16T02:55:26.691 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45191.log 2024-09-16T02:55:26.691 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62309.log: /var/log/ceph/ceph-client.admin.67303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62309.log.gz 2024-09-16T02:55:26.691 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67303.log.gz 2024-09-16T02:55:26.691 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45477.log 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43549.log 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45477.log: /var/log/ceph/ceph-client.admin.45191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45191.log.gz 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45477.log.gz 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32670.log 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43315.log 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43549.log: /var/log/ceph/ceph-client.admin.32670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43549.log.gz 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32670.log.gz 2024-09-16T02:55:26.693 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47089.log 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66211.log 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43315.log: /var/log/ceph/ceph-client.admin.47089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43315.log.gz 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47089.log.gz 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47428.log 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48963.log 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66211.log: /var/log/ceph/ceph-client.admin.47428.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66211.log.gz 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47428.log.gz 2024-09-16T02:55:26.695 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36104.log 2024-09-16T02:55:26.697 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69409.log 2024-09-16T02:55:26.697 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48963.log.gz 2024-09-16T02:55:26.697 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36104.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48521.log 2024-09-16T02:55:26.697 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36104.log.gz 2024-09-16T02:55:26.697 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69409.log.gz 2024-09-16T02:55:26.697 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54034.log 2024-09-16T02:55:26.697 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64233.log 2024-09-16T02:55:26.698 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48521.log: /var/log/ceph/ceph-client.admin.54034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48521.log.gz 2024-09-16T02:55:26.699 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54034.log.gz 2024-09-16T02:55:26.699 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49405.log 2024-09-16T02:55:26.699 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64233.log.gz 2024-09-16T02:55:26.700 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65951.log 2024-09-16T02:55:26.700 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67849.log 2024-09-16T02:55:26.700 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49405.log: /var/log/ceph/ceph-client.admin.65951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49405.log.gz 2024-09-16T02:55:26.700 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65951.log.gz 2024-09-16T02:55:26.700 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46725.log 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52188.log 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67849.log: /var/log/ceph/ceph-client.admin.46725.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67849.log.gz 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46725.log.gz 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40031.log 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53020.log 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52188.log: /var/log/ceph/ceph-client.admin.40031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52188.log.gz 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40031.log.gz 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46777.log 2024-09-16T02:55:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61242.log 2024-09-16T02:55:26.704 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53020.log.gz 2024-09-16T02:55:26.704 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46777.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46777.log.gz 2024-09-16T02:55:26.704 INFO:teuthology.orchestra.run.smithi137.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68785.log 2024-09-16T02:55:26.704 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61242.log.gz 2024-09-16T02:55:26.705 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42222.log 2024-09-16T02:55:26.705 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42768.log 2024-09-16T02:55:26.705 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68785.log: /var/log/ceph/ceph-client.admin.42222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68785.log.gz 2024-09-16T02:55:26.705 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42222.log.gz 2024-09-16T02:55:26.705 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34440.log 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42768.log.gz 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42014.log 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48313.log 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34440.log.gz 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48937.log 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42014.log.gz 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48313.log.gz 2024-09-16T02:55:26.707 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53748.log 2024-09-16T02:55:26.709 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41284.log 2024-09-16T02:55:26.709 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48937.log.gz 2024-09-16T02:55:26.709 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53748.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32302.log 2024-09-16T02:55:26.709 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53748.log.gz 2024-09-16T02:55:26.709 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41284.log.gz 2024-09-16T02:55:26.709 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36546.log 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56094.log 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32302.log: /var/log/ceph/ceph-client.admin.36546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32302.log.gz 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36546.log.gz 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36676.log 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67095.log 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56094.log: /var/log/ceph/ceph-client.admin.36676.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56094.log.gz 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36676.log.gz 2024-09-16T02:55:26.711 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35480.log 2024-09-16T02:55:26.713 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52344.log 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67095.log: /var/log/ceph/ceph-client.admin.35480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67095.log.gz 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35480.log.gz 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43523.log 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52344.log: /var/log/ceph/ceph-client.admin.43523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52344.log.gz 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43523.log.gz 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63297.log 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44408.log 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47532.log: /var/log/ceph/ceph-client.admin.63297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63297.log.gz 2024-09-16T02:55:26.714 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37196.log 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32134.log 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44408.log: /var/log/ceph/ceph-client.admin.37196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44408.log.gz 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37196.log.gz 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40473.log 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55678.log 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32134.log: /var/log/ceph/ceph-client.admin.40473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32134.log.gz 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40473.log.gz 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66081.log 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67641.log 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55678.log: /var/log/ceph/ceph-client.admin.66081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55678.log.gz 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66081.log.gz 2024-09-16T02:55:26.717 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49821.log 2024-09-16T02:55:26.720 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61866.log 2024-09-16T02:55:26.720 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67641.log: /var/log/ceph/ceph-client.admin.49821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67641.log.gz 2024-09-16T02:55:26.720 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49821.log.gz 2024-09-16T02:55:26.720 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54919.log 2024-09-16T02:55:26.720 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61580.log 2024-09-16T02:55:26.720 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61866.log: /var/log/ceph/ceph-client.admin.54919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61866.log.gz 2024-09-16T02:55:26.721 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54919.log.gz 2024-09-16T02:55:26.721 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40005.log 2024-09-16T02:55:26.721 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70424.log 2024-09-16T02:55:26.721 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61580.log.gz 2024-09-16T02:55:26.721 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40005.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 2024-09-16T02:55:26.721 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.721 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70424.log.gz 2024-09-16T02:55:26.723 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57160.log 2024-09-16T02:55:26.723 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43002.log 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52942.log: /var/log/ceph/ceph-client.admin.57160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57160.log.gz 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61892.log 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67251.log 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43002.log: /var/log/ceph/ceph-client.admin.61892.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61892.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43002.log.gz 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65613.log 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35220.log 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67251.log: /var/log/ceph/ceph-client.admin.65613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67251.log.gz 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65613.log.gz 2024-09-16T02:55:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47636.log 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57186.log 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35220.log: /var/log/ceph/ceph-client.admin.47636.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35220.log.gz 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47636.log.gz 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61268.log 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69644.log 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57186.log: /var/log/ceph/ceph-client.admin.61268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61268.log.gz 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57186.log.gz 2024-09-16T02:55:26.726 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55886.log 2024-09-16T02:55:26.728 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42898.log 2024-09-16T02:55:26.728 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55886.log: /var/log/ceph/ceph-client.admin.69644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69644.log.gz 2024-09-16T02:55:26.728 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55886.log.gz 2024-09-16T02:55:26.728 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37482.log 2024-09-16T02:55:26.728 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35168.log 2024-09-16T02:55:26.728 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42898.log: /var/log/ceph/ceph-client.admin.37482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42898.log.gz 2024-09-16T02:55:26.729 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37482.log.gz 2024-09-16T02:55:26.729 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51902.log 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35168.log: /var/log/ceph/ceph-client.admin.51902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35168.log.gz 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51902.log.gz 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42586.log 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38106.log 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42586.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39094.log 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42586.log.gz 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38106.log.gz 2024-09-16T02:55:26.731 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47896.log 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64598.log 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39094.log.gz 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47896.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55361.log 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47896.log.gz 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64598.log.gz 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56848.log 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34024.log 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55361.log.gz 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56848.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51564.log 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56848.log.gz 2024-09-16T02:55:26.734 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34024.log.gz 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70528.log 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68265.log 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51564.log: /var/log/ceph/ceph-client.admin.70528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51564.log.gz 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70528.log.gz 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63531.log 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57403.log 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68265.log.gz 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63531.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47167.log 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63531.log.gz 2024-09-16T02:55:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57403.log.gz 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64155.log 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52968.log 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47167.log: /var/log/ceph/ceph-client.admin.64155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47167.log.gz 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64155.log.gz 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41570.log 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52968.log.gz 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41570.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36468.log 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41570.log.gz 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-09-16T02:55:26.740 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64650.log 2024-09-16T02:55:26.742 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33060.log 2024-09-16T02:55:26.742 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36468.log.gz 2024-09-16T02:55:26.742 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64650.log.gz 2024-09-16T02:55:26.742 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38340.log 2024-09-16T02:55:26.742 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33060.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40369.log 2024-09-16T02:55:26.742 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33060.log.gz 2024-09-16T02:55:26.742 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38340.log.gz 2024-09-16T02:55:26.744 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38626.log 2024-09-16T02:55:26.744 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48547.log 2024-09-16T02:55:26.744 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40369.log.gz 2024-09-16T02:55:26.745 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38626.log.gz 2024-09-16T02:55:26.745 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46309.log 2024-09-16T02:55:26.745 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61996.log 2024-09-16T02:55:26.745 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48547.log: /var/log/ceph/ceph-client.admin.46309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48547.log.gz 2024-09-16T02:55:26.745 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46309.log.gz 2024-09-16T02:55:26.745 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-09-16T02:55:26.747 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44330.log 2024-09-16T02:55:26.747 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61996.log: /var/log/ceph/ceph-client.admin.39589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61996.log.gz 2024-09-16T02:55:26.747 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-09-16T02:55:26.747 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44147.log 2024-09-16T02:55:26.747 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32297.log 2024-09-16T02:55:26.747 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44330.log.gz 2024-09-16T02:55:26.747 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44147.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45373.log 2024-09-16T02:55:26.748 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44147.log.gz 2024-09-16T02:55:26.748 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32297.log.gz 2024-09-16T02:55:26.748 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56796.log 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69123.log 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45373.log: /var/log/ceph/ceph-client.admin.56796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45373.log.gz 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56796.log.gz 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38028.log 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56900.log 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69123.log.gz 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38028.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56042.log 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.750 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56900.log.gz 2024-09-16T02:55:26.751 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54268.log 2024-09-16T02:55:26.752 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37066.log 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56042.log: /var/log/ceph/ceph-client.admin.54268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56042.log.gz 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54268.log.gz 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31928.log 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37066.log.gz 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41622.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.31718.log 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31928.log.gz 2024-09-16T02:55:26.753 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70665.log 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62491.log 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31718.log: /var/log/ceph/ceph-client.admin.70665.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31718.log.gz 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70665.log.gz 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66549.log 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31827.log 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62491.log: /var/log/ceph/ceph-client.admin.66549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62491.log.gz 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66549.log.gz 2024-09-16T02:55:26.755 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67173.log 2024-09-16T02:55:26.757 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26459.log 2024-09-16T02:55:26.757 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31827.log: /var/log/ceph/ceph-client.admin.67173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31827.log.gz 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67173.log.gz 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48287.log 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48911.log 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26459.log: /var/log/ceph/ceph-client.admin.48287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26459.log.gz 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48287.log.gz 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33190.log 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44252.log 2024-09-16T02:55:26.758 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48911.log.gz 2024-09-16T02:55:26.760 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50263.log 2024-09-16T02:55:26.760 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33190.log.gz 2024-09-16T02:55:26.760 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44252.log.gz 2024-09-16T02:55:26.761 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-09-16T02:55:26.761 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66237.log 2024-09-16T02:55:26.761 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50263.log.gz 2024-09-16T02:55:26.761 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39276.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48182.log 2024-09-16T02:55:26.761 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-09-16T02:55:26.761 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66237.log.gz 2024-09-16T02:55:26.761 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49925.log 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43497.log 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48182.log.gz 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49925.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53800.log 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49925.log.gz 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43497.log.gz 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61554.log 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65717.log 2024-09-16T02:55:26.763 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53800.log: /var/log/ceph/ceph-client.admin.61554.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61554.log.gz 2024-09-16T02:55:26.764 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53800.log.gz 2024-09-16T02:55:26.764 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69019.log 2024-09-16T02:55:26.765 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43809.log 2024-09-16T02:55:26.765 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65717.log.gz 2024-09-16T02:55:26.765 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-09-16T02:55:26.765 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69019.log: /var/log/ceph/ceph-client.admin.43809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69019.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43809.log.gz 2024-09-16T02:55:26.765 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:55:26.766 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33710.log 2024-09-16T02:55:26.766 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60930.log 2024-09-16T02:55:26.766 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48859.log 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33710.log: /var/log/ceph/ceph-client.admin.60930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60930.log.gz 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33710.log.gz 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63375.log 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48859.log.gz 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46985.log 2024-09-16T02:55:26.768 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57108.log 2024-09-16T02:55:26.770 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63375.log.gz 2024-09-16T02:55:26.770 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46985.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50705.log 2024-09-16T02:55:26.770 INFO:teuthology.orchestra.run.smithi137.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46985.log.gz 2024-09-16T02:55:26.770 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57108.log.gz 2024-09-16T02:55:26.770 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T02:55:26.770 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64025.log 2024-09-16T02:55:26.770 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50705.log.gz 2024-09-16T02:55:26.771 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68967.log 2024-09-16T02:55:26.772 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64025.log.gz 2024-09-16T02:55:26.772 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55153.log 2024-09-16T02:55:26.772 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68967.log.gz 2024-09-16T02:55:26.772 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53306.log 2024-09-16T02:55:26.772 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55153.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55153.log.gz 2024-09-16T02:55:26.773 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37534.log 2024-09-16T02:55:26.773 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53306.log.gz 2024-09-16T02:55:26.773 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62178.log 2024-09-16T02:55:26.774 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37534.log.gz 2024-09-16T02:55:26.775 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26483.log 2024-09-16T02:55:26.775 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62178.log.gz 2024-09-16T02:55:26.775 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43679.log 2024-09-16T02:55:26.775 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26483.log.gz 2024-09-16T02:55:26.776 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55756.log 2024-09-16T02:55:26.776 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43679.log.gz 2024-09-16T02:55:26.777 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31937.log 2024-09-16T02:55:26.777 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55756.log.gz 2024-09-16T02:55:26.778 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62959.log 2024-09-16T02:55:26.778 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31937.log.gz 2024-09-16T02:55:26.779 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39302.log 2024-09-16T02:55:26.779 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62959.log.gz 2024-09-16T02:55:26.780 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36832.log 2024-09-16T02:55:26.780 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39302.log.gz 2024-09-16T02:55:26.781 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2024-09-16T02:55:26.781 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36832.log.gz 2024-09-16T02:55:26.782 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33580.log 2024-09-16T02:55:26.783 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2024-09-16T02:55:26.783 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43783.log 2024-09-16T02:55:26.784 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33580.log.gz 2024-09-16T02:55:26.784 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51095.log 2024-09-16T02:55:26.785 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43783.log.gz 2024-09-16T02:55:26.785 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2024-09-16T02:55:26.786 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51095.log.gz 2024-09-16T02:55:26.786 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-09-16T02:55:26.787 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2024-09-16T02:55:26.787 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51069.log 2024-09-16T02:55:26.788 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-09-16T02:55:26.788 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55912.log 2024-09-16T02:55:26.789 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51069.log.gz 2024-09-16T02:55:26.789 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37118.log 2024-09-16T02:55:26.790 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55912.log.gz 2024-09-16T02:55:26.790 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69878.log 2024-09-16T02:55:26.791 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37118.log.gz 2024-09-16T02:55:26.791 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38704.log 2024-09-16T02:55:26.792 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69878.log.gz 2024-09-16T02:55:26.792 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49613.log 2024-09-16T02:55:26.793 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38704.log.gz 2024-09-16T02:55:26.793 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46543.log 2024-09-16T02:55:26.794 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49613.log.gz 2024-09-16T02:55:26.794 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40551.log 2024-09-16T02:55:26.795 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46543.log.gz 2024-09-16T02:55:26.795 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45165.log 2024-09-16T02:55:26.796 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40551.log.gz 2024-09-16T02:55:26.796 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53826.log 2024-09-16T02:55:26.797 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45165.log.gz 2024-09-16T02:55:26.797 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47193.log 2024-09-16T02:55:26.798 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53826.log.gz 2024-09-16T02:55:26.798 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42248.log 2024-09-16T02:55:26.799 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.47193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47193.log.gz 2024-09-16T02:55:26.799 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32804.log 2024-09-16T02:55:26.800 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42248.log.gz 2024-09-16T02:55:26.800 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46881.log 2024-09-16T02:55:26.801 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32804.log.gz 2024-09-16T02:55:26.801 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26402.log 2024-09-16T02:55:26.802 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46881.log.gz 2024-09-16T02:55:26.802 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36988.log 2024-09-16T02:55:26.803 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26402.log.gz 2024-09-16T02:55:26.803 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67355.log 2024-09-16T02:55:26.804 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36988.log.gz 2024-09-16T02:55:26.805 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42274.log 2024-09-16T02:55:26.805 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67355.log.gz 2024-09-16T02:55:26.806 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44278.log 2024-09-16T02:55:26.806 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42274.log.gz 2024-09-16T02:55:26.807 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44668.log 2024-09-16T02:55:26.807 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44278.log.gz 2024-09-16T02:55:26.808 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36858.log 2024-09-16T02:55:26.808 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44668.log.gz 2024-09-16T02:55:26.809 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65197.log 2024-09-16T02:55:26.809 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36858.log.gz 2024-09-16T02:55:26.810 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37378.log 2024-09-16T02:55:26.810 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65197.log.gz 2024-09-16T02:55:26.811 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 2024-09-16T02:55:26.811 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37378.log.gz 2024-09-16T02:55:26.812 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2024-09-16T02:55:26.812 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gz 2024-09-16T02:55:26.813 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43991.log 2024-09-16T02:55:26.813 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2024-09-16T02:55:26.814 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63453.log 2024-09-16T02:55:26.814 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43991.log.gz 2024-09-16T02:55:26.815 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68759.log 2024-09-16T02:55:26.816 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.63453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63453.log.gz 2024-09-16T02:55:26.816 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65405.log 2024-09-16T02:55:26.817 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68759.log.gz 2024-09-16T02:55:26.817 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38678.log 2024-09-16T02:55:26.822 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65405.log.gz 2024-09-16T02:55:26.822 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52058.log 2024-09-16T02:55:26.822 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38678.log.gz 2024-09-16T02:55:26.822 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48677.log 2024-09-16T02:55:26.823 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52058.log.gz 2024-09-16T02:55:26.823 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38912.log 2024-09-16T02:55:26.823 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.48677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48677.log.gz 2024-09-16T02:55:26.823 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56250.log 2024-09-16T02:55:26.823 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38912.log.gz 2024-09-16T02:55:26.823 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61528.log 2024-09-16T02:55:26.824 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56250.log.gz 2024-09-16T02:55:26.824 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55730.log 2024-09-16T02:55:26.824 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61528.log.gz 2024-09-16T02:55:26.824 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51876.log 2024-09-16T02:55:26.825 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55730.log.gz 2024-09-16T02:55:26.825 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36312.log 2024-09-16T02:55:26.826 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51876.log.gz 2024-09-16T02:55:26.826 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68161.log 2024-09-16T02:55:26.827 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36312.log.gz 2024-09-16T02:55:26.827 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52708.log 2024-09-16T02:55:26.828 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68161.log.gz 2024-09-16T02:55:26.828 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31930.log 2024-09-16T02:55:26.829 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52708.log.gz 2024-09-16T02:55:26.829 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68447.log 2024-09-16T02:55:26.830 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31930.log.gz 2024-09-16T02:55:26.830 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42144.log 2024-09-16T02:55:26.831 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68447.log.gz 2024-09-16T02:55:26.831 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-09-16T02:55:26.832 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42144.log.gz 2024-09-16T02:55:26.832 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49639.log 2024-09-16T02:55:26.833 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-09-16T02:55:26.833 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42638.log 2024-09-16T02:55:26.834 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49639.log.gz 2024-09-16T02:55:26.835 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40733.log 2024-09-16T02:55:26.835 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42638.log.gz 2024-09-16T02:55:26.835 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31938.log 2024-09-16T02:55:26.836 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40733.log.gz 2024-09-16T02:55:26.837 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39172.log 2024-09-16T02:55:26.837 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31938.log.gz 2024-09-16T02:55:26.838 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38600.log 2024-09-16T02:55:26.838 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39172.log.gz 2024-09-16T02:55:26.839 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44720.log 2024-09-16T02:55:26.839 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38600.log.gz 2024-09-16T02:55:26.840 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56198.log 2024-09-16T02:55:26.840 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44720.log.gz 2024-09-16T02:55:26.841 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37508.log 2024-09-16T02:55:26.841 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56198.log.gz 2024-09-16T02:55:26.842 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68889.log 2024-09-16T02:55:26.842 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37508.log.gz 2024-09-16T02:55:26.843 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58294.log 2024-09-16T02:55:26.843 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68889.log.gz 2024-09-16T02:55:26.844 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65431.log 2024-09-16T02:55:26.844 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58294.log.gz 2024-09-16T02:55:26.845 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64729.log 2024-09-16T02:55:26.846 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65431.log.gz 2024-09-16T02:55:26.846 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36338.log 2024-09-16T02:55:26.847 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64729.log.gz 2024-09-16T02:55:26.847 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31770.log 2024-09-16T02:55:26.848 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36338.log.gz 2024-09-16T02:55:26.848 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34934.log 2024-09-16T02:55:26.849 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.31770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31770.log.gz 2024-09-16T02:55:26.849 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51303.log 2024-09-16T02:55:26.850 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34934.log.gz 2024-09-16T02:55:26.850 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50315.log 2024-09-16T02:55:26.851 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51303.log.gz 2024-09-16T02:55:26.851 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61658.log 2024-09-16T02:55:26.852 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50315.log.gz 2024-09-16T02:55:26.852 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26656.log 2024-09-16T02:55:26.853 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61658.log.gz 2024-09-16T02:55:26.853 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56692.log 2024-09-16T02:55:26.854 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.26656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26656.log.gz 2024-09-16T02:55:26.854 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67745.log 2024-09-16T02:55:26.855 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56692.log.gz 2024-09-16T02:55:26.855 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33242.log 2024-09-16T02:55:26.856 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67745.log.gz 2024-09-16T02:55:26.856 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39901.log 2024-09-16T02:55:26.857 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33242.log.gz 2024-09-16T02:55:26.857 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66731.log 2024-09-16T02:55:26.858 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39901.log.gz 2024-09-16T02:55:26.858 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-09-16T02:55:26.859 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66731.log.gz 2024-09-16T02:55:26.859 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33920.log 2024-09-16T02:55:26.860 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-09-16T02:55:26.860 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41258.log 2024-09-16T02:55:26.861 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33920.log.gz 2024-09-16T02:55:26.861 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52734.log 2024-09-16T02:55:26.862 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41258.log.gz 2024-09-16T02:55:26.862 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56328.log 2024-09-16T02:55:26.863 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52734.log.gz 2024-09-16T02:55:26.863 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57513.log 2024-09-16T02:55:26.864 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56328.log.gz 2024-09-16T02:55:26.864 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51459.log 2024-09-16T02:55:26.865 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57513.log.gz 2024-09-16T02:55:26.866 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44486.log 2024-09-16T02:55:26.866 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51459.log.gz 2024-09-16T02:55:26.867 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38834.log 2024-09-16T02:55:26.867 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44486.log.gz 2024-09-16T02:55:26.868 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65067.log 2024-09-16T02:55:26.868 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38834.log.gz 2024-09-16T02:55:26.869 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46491.log 2024-09-16T02:55:26.869 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65067.log.gz 2024-09-16T02:55:26.870 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44772.log 2024-09-16T02:55:26.870 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46491.log.gz 2024-09-16T02:55:26.871 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58194.log 2024-09-16T02:55:26.871 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44772.log.gz 2024-09-16T02:55:26.872 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44304.log 2024-09-16T02:55:26.872 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.58194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58194.log.gz 2024-09-16T02:55:26.873 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66861.log 2024-09-16T02:55:26.873 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44304.log.gz 2024-09-16T02:55:26.874 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52136.log 2024-09-16T02:55:26.874 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66861.log.gz 2024-09-16T02:55:26.875 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32405.log 2024-09-16T02:55:26.875 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52136.log.gz 2024-09-16T02:55:26.876 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61788.log 2024-09-16T02:55:26.877 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32405.log.gz 2024-09-16T02:55:26.877 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50341.log 2024-09-16T02:55:26.878 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61788.log.gz 2024-09-16T02:55:26.878 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57469.log 2024-09-16T02:55:26.879 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50341.log.gz 2024-09-16T02:55:26.879 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32160.log 2024-09-16T02:55:26.880 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57469.log.gz 2024-09-16T02:55:26.880 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54736.log 2024-09-16T02:55:26.881 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.32160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32160.log.gz 2024-09-16T02:55:26.881 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34674.log 2024-09-16T02:55:26.882 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.54736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54736.log.gz 2024-09-16T02:55:26.882 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61476.log 2024-09-16T02:55:26.883 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34674.log.gz 2024-09-16T02:55:26.883 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66419.log 2024-09-16T02:55:26.884 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.61476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61476.log.gz 2024-09-16T02:55:26.884 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55442.log 2024-09-16T02:55:26.885 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66419.log.gz 2024-09-16T02:55:26.885 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33946.log 2024-09-16T02:55:26.886 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.55442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55442.log.gz 2024-09-16T02:55:26.886 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42534.log 2024-09-16T02:55:26.887 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.33946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33946.log.gz 2024-09-16T02:55:26.887 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52162.log 2024-09-16T02:55:26.888 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.42534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42534.log.gz 2024-09-16T02:55:26.888 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-09-16T02:55:26.889 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52162.log.gz 2024-09-16T02:55:26.889 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50913.log 2024-09-16T02:55:26.890 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.34492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-09-16T02:55:26.890 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65873.log 2024-09-16T02:55:26.891 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50913.log.gz 2024-09-16T02:55:26.891 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41336.log 2024-09-16T02:55:26.892 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65873.log.gz 2024-09-16T02:55:26.892 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2024-09-16T02:55:26.893 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41336.log.gz 2024-09-16T02:55:26.893 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65795.log 2024-09-16T02:55:26.894 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.39380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2024-09-16T02:55:26.894 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51798.log 2024-09-16T02:55:26.895 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65795.log.gz 2024-09-16T02:55:26.895 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37248.log 2024-09-16T02:55:26.896 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.51798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2024-09-16T02:55:26.897 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69592.log 2024-09-16T02:55:26.897 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37248.log.gz 2024-09-16T02:55:26.898 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57645.log 2024-09-16T02:55:26.898 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69592.log.gz 2024-09-16T02:55:26.899 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56276.log 2024-09-16T02:55:26.899 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57645.log.gz 2024-09-16T02:55:26.900 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-09-16T02:55:26.900 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.56276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56276.log.gz 2024-09-16T02:55:26.901 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43263.log 2024-09-16T02:55:26.901 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.40837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-09-16T02:55:26.902 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43861.log 2024-09-16T02:55:26.903 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43263.log.gz 2024-09-16T02:55:26.903 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65093.log 2024-09-16T02:55:26.903 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43861.log.gz 2024-09-16T02:55:26.904 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38158.log 2024-09-16T02:55:26.905 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.65093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65093.log.gz 2024-09-16T02:55:26.905 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52994.log 2024-09-16T02:55:26.906 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.38158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38158.log.gz 2024-09-16T02:55:26.906 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66679.log 2024-09-16T02:55:26.907 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52994.log.gz 2024-09-16T02:55:26.907 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37326.log 2024-09-16T02:55:26.908 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.66679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66679.log.gz 2024-09-16T02:55:26.908 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50653.log 2024-09-16T02:55:26.909 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.37326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37326.log.gz 2024-09-16T02:55:26.909 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68135.log 2024-09-16T02:55:26.910 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.50653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50653.log.gz 2024-09-16T02:55:26.910 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57696.log 2024-09-16T02:55:26.911 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68135.log.gz 2024-09-16T02:55:26.911 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52812.log 2024-09-16T02:55:26.912 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57696.log.gz 2024-09-16T02:55:26.912 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69540.log 2024-09-16T02:55:26.913 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.52812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52812.log.gz 2024-09-16T02:55:26.913 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43731.log 2024-09-16T02:55:26.914 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.69540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69540.log.gz 2024-09-16T02:55:26.914 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64311.log 2024-09-16T02:55:26.915 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43731.log.gz 2024-09-16T02:55:26.915 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68395.log 2024-09-16T02:55:26.916 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64311.log.gz 2024-09-16T02:55:26.916 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35272.log 2024-09-16T02:55:26.917 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68395.log.gz 2024-09-16T02:55:26.917 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57030.log 2024-09-16T02:55:26.918 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.35272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35272.log.gz 2024-09-16T02:55:26.918 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62283.log 2024-09-16T02:55:26.919 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.57030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57030.log.gz 2024-09-16T02:55:26.919 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67069.log 2024-09-16T02:55:26.920 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.62283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62283.log.gz 2024-09-16T02:55:26.920 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44434.log 2024-09-16T02:55:26.921 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67069.log.gz 2024-09-16T02:55:26.921 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53358.log 2024-09-16T02:55:26.922 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.44434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44434.log.gz 2024-09-16T02:55:26.922 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-09-16T02:55:26.923 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.53358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53358.log.gz 2024-09-16T02:55:26.924 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46907.log 2024-09-16T02:55:26.924 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.45217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-09-16T02:55:26.925 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36910.log 2024-09-16T02:55:26.925 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46907.log.gz 2024-09-16T02:55:26.926 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49275.log 2024-09-16T02:55:26.926 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36910.log.gz 2024-09-16T02:55:26.927 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70034.log 2024-09-16T02:55:26.927 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.49275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49275.log.gz 2024-09-16T02:55:26.928 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64494.log 2024-09-16T02:55:26.928 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.70034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70034.log.gz 2024-09-16T02:55:26.929 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68473.log 2024-09-16T02:55:26.929 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.64494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64494.log.gz 2024-09-16T02:55:26.930 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46387.log 2024-09-16T02:55:26.930 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.68473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68473.log.gz 2024-09-16T02:55:26.931 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41674.log 2024-09-16T02:55:26.931 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.46387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46387.log.gz 2024-09-16T02:55:26.932 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41648.log 2024-09-16T02:55:26.932 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41674.log.gz 2024-09-16T02:55:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21483.log 2024-09-16T02:55:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.41648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41648.log.gz 2024-09-16T02:55:26.934 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67147.log 2024-09-16T02:55:26.934 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.tmp-client.admin.21483.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21483.log.gz 2024-09-16T02:55:26.935 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43757.log 2024-09-16T02:55:26.936 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.67147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67147.log.gz 2024-09-16T02:55:26.936 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.43757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43757.log.gz 2024-09-16T02:55:27.450 INFO:teuthology.orchestra.run.smithi137.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T02:55:27.719 INFO:teuthology.orchestra.run.smithi137.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T02:55:27.749 INFO:teuthology.orchestra.run.smithi137.stderr: 89.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T02:55:38.607 INFO:teuthology.orchestra.run.smithi137.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T02:55:43.268 INFO:teuthology.orchestra.run.smithi137.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T02:55:43.929 INFO:teuthology.orchestra.run.smithi137.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T02:58:20.482 INFO:teuthology.orchestra.run.smithi137.stderr: 90.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T02:58:20.483 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-16T02:58:20.484 INFO:teuthology.orchestra.run.smithi137.stderr:real 2m54.369s 2024-09-16T02:58:20.484 INFO:teuthology.orchestra.run.smithi137.stderr:user 3m38.512s 2024-09-16T02:58:20.484 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m7.572s 2024-09-16T02:58:20.484 INFO:tasks.ceph:Archiving logs... 2024-09-16T02:58:20.485 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763/remote/smithi137/log 2024-09-16T02:58:20.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T02:58:57.855 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T02:58:57.872 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T02:58:57.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T02:58:57.910 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T02:58:57.919 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T02:58:57.919 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:58:58.017 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:58:58.017 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2024-09-16T02:58:58.018 INFO:teuthology.orchestra.run.smithi137.stdout:+hv01.front.sepi 67.205.162.81 3 u 92 128 377 0.087 -4.085 1.948 2024-09-16T02:58:58.018 INFO:teuthology.orchestra.run.smithi137.stdout:*hv02.front.sepi 96.245.170.99 3 u 93 128 377 0.092 -4.144 1.232 2024-09-16T02:58:58.018 INFO:teuthology.orchestra.run.smithi137.stdout:+hv03.front.sepi 74.6.168.72 3 u 88 128 377 0.089 -2.365 1.857 2024-09-16T02:58:58.018 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-16T02:58:58.019 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T02:58:58.031 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T02:58:58.032 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T02:58:58.041 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T02:58:58.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T02:58:58.065 INFO:teuthology.task.internal:Duration was 2378.511930 seconds 2024-09-16T02:58:58.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T02:58:58.074 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T02:58:58.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T02:58:58.212 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T02:58:58.212 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-09-16T02:58:58.212 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T02:58:58.223 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T02:58:58.223 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T02:58:58.304 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T02:58:58.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T02:58:58.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T02:58:58.490 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T02:58:58.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T02:58:58.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T02:58:58.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:58:58.602 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-09-16T02:58:58.634 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:58:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:58:58.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T02:58:58.690 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T02:58:58.690 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905763/remote/smithi137 2024-09-16T02:58:58.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T02:58:58.744 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T02:58:58.744 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T02:58:58.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T02:58:58.850 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T02:58:58.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T02:58:59.524 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T02:58:59.525 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T02:58:59.531 INFO:teuthology.orchestra.run.smithi137.stdout: 655475 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 02:58 /home/ubuntu/cephtest 2024-09-16T02:58:59.533 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T02:59:00.256 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T02:59:00.714 INFO:teuthology.run:Summary data: description: rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest}} duration: 2378.5119297504425 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T02:59:00.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T02:59:00.802 INFO:teuthology.run:pass